Sunday, December 30, 2007

Useful UNIX commands for WebSphere & Portal Administrator

These are some useful commands that i came across when administrating WebSphere Application Server or WebSphere Portal Server. Note these commands are specifically written for LINUX but most of them would for other UNIX platforms like Solaris, AIX and other flavours of LINUX like SUSE or REDHAT.

Performance related Commands to monitor system performance
-------------------------------------------------------------------------
1) Find the process that uses most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -2
(eg) output of the above command when websphere server process is utilizing the most of the CPU





%CPU   PID USER     COMMAND
71.7 31237 root /cust/IBM/WebSphere/AppServer/java/bin/java
-Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere

/AppServer/java/jre/lib/ext/ibmext.jar
-Dwas.status.socket=54859 -classpath

/cust/IBM/WebSphere/AppServer/profiles/AppSrv02/properties:/cust/IBM/WebSphere/AppSer



2) Find the last 10 process that use the most CPU
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10





%CPU   PID USER     COMMAND
2.1 6375 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=2132 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.3 13886 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=25042 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer
0.2 17271 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.2 17871 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9606 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.2 17713 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=9140 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppServ
0.1 18499 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=11309 -classpath /cust/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/cust/IBM/WebSphere/AppSer
0.1 17504 root /cust/IBM/WebSphere/AppServer/java/bin/java -Xbootclasspath/p:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/cust/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -Dwas.status.socket=8969 -classpath /cust/IBM/WebSphere/AppServer/profiles/Dmgr01/properties:/cust/IBM/WebSphere/AppServer


3)Find cpu usage
mpstat -P ALL or mpstat





Linux 2.4.21-27.ELsmp (machine1.www.com)   02/02/2008

11:55:00 AM CPU %user %nice %system %iowait %irq %soft %idle intr/s
11:55:00 AM all 1.51 0.00 0.41 0.04 0.01 0.09 97.94 184.92
11:55:00 AM 0 1.50 0.00 0.41 0.04 0.01 0.11 97.93 123.76
11:55:00 AM 1 1.51 0.00 0.42 0.04 0.00 0.07 97.95 61.16


4)Find CPU usage with the process listing in runtime and much more statistics in terms of memory, virtual memory, etc.
top






12:00:20 up 118 days, 15:16, 2 users, load average: 0.07, 0.02, 0.00
572 processes: 570 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 2.7% 0.0% 1.1% 0.0% 0.0% 0.0% 95.9%
cpu00 1.9% 0.0% 0.3% 0.0% 0.0% 0.0% 97.6%
cpu01 3.5% 0.0% 1.9% 0.1% 0.0% 0.0% 94.2%
Mem: 11819592k av, 5088092k used, 6731500k free, 0k shrd, 295304k buff
3998132k active, 682312k inactive
Swap: 2048276k av, 0k used, 2048276k free 1258592k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
17719 root 16 0 306M 306M 3992 S 0.7 2.6 22:14 1 java
19358 root 16 0 2532 2532 896 R 0.5 0.0 0:00 1 top
17771 root 15 0 306M 306M 3992 S 0.4 2.6 3:48 0 java
18058 root 16 0 458M 458M 4032 S 0.1 3.9 12:55 0 java
1 root 15 0 512 512 452 S 0.0 0.0 1:19 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0


5)Continuous CPU usage monitoring and find the average CPU used.
a) sar -u 2 5
b) sar -o output.file 12 8 >/dev/null 2>&1 &
c) nohup sar -o output.file 12 8 >/dev/null 2>&1 &






Linux 2.4.21-27.ELsmp (machine1.web.com)   02/02/2008

12:04:34 PM CPU %user %nice %system %iowait %idle
12:04:36 PM all 0.25 0.00 0.00 0.00 99.75
12:04:38 PM all 0.00 0.00 0.50 0.25 99.25
12:04:40 PM all 7.00 0.00 7.25 0.00 85.75
12:04:42 PM all 21.05 0.00 1.50 0.00 77.44
12:04:44 PM all 3.00 0.00 0.50 0.00 96.50
Average: all 6.26 0.00 1.95 0.05 91.74



Disk Usage related Commands to monitor Disk space
--------------------------------------------------------
1)Disk Usage on all mounts
df -h -T






Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext3 2.9G 785M 2.0G 29% /
/dev/sda7 ext3 57G 9.3G 45G 18% /cust
/dev/sda6 ext3 2.0G 82M 1.8G 5% /lc
none tmpfs 5.7G 0 5.7G 0% /dev/shm
/dev/sda5 ext3 2.0G 33M 1.8G 2% /tmp
/dev/sda3 ext3 2.0G 64M 1.8G 4% /var


2)prints just one line with the total size of the directory, the below example show the size of the /usr/IBM/Websphere WebSphere Portal Server directory.
du -ch | grep total






bash-2.05b# du -ch | grep total
6.3G total



Network related Commands
-------------------------------------
1)
a) dig (domain information groper) - is a flexible tool for interrogating DNS name servers.
b) nslookup - Queries a name server for a host or domain lookup.
c) host - DNS lookup uility.

The below example shows the information like CNAME , Aliases, ipaddres , etc about www.google.com


a) dig <ip address or hostname>







bash-2.05b# dig www.google.com

; <<>> DiG 9.2.4 <<>> www.google.com
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1088
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 7, ADDITIONAL: 0

;; QUESTION SECTION:
;www.google.com. IN A

;; ANSWER SECTION:
www.google.com. 585119 IN CNAME www.l.google.com.
www.l.google.com. 69 IN A 216.239.51.99
www.l.google.com. 69 IN A 216.239.51.104

;; AUTHORITY SECTION:
l.google.com. 60734 IN NS g.l.google.com.
l.google.com. 60734 IN NS b.l.google.com.
l.google.com. 60734 IN NS f.l.google.com.
l.google.com. 60734 IN NS c.l.google.com.
l.google.com. 60734 IN NS a.l.google.com.
l.google.com. 60734 IN NS e.l.google.com.
l.google.com. 60734 IN NS d.l.google.com.

;; Query time: 2 msec
;; SERVER: 10.5.50.184#53(10.5.50.184)
;; WHEN: Sun Feb 3 09:51:46 2008
;; MSG SIZE rcvd: 196


b) host <ip address or hostname>






bash-2.05b# host www.google.com
www.google.com is an alias for www.l.google.com.
www.l.google.com has address 216.239.51.104
www.l.google.com has address 216.239.51.99


c) nslookup <ip address or hostname>






bash-2.05b# nslookup www.google.com
Server: 10.5.50.184
Address: 10.5.50.184#53

Non-authoritative answer:
www.google.com canonical name = www.l.google.com.
Name: www.l.google.com
Address: 216.239.51.99
Name: www.l.google.com
Address: 216.239.51.104


2) List all network interfaces, the below example list two interfaces eth0 and eth1 and a loopback.


infconfig -a







bash-2.05b# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:11:43:32:49:99
inet addr:10.2.150.74 Bcast:10.5.50.127 Mask:255.255.255.192
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:357854361 errors:0 dropped:0 overruns:0 frame:0
TX packets:383621806 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3168219417 (3021.4 Mb) TX bytes:3724305382 (3551.7 Mb)
Interrupt:17

eth1 Link encap:Ethernet HWaddr 00:11:43:32:49:9A
inet addr:10.5.154.138 Bcast:10.5.54.127 Mask:255.255.255.128
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:18

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:92809439 errors:0 dropped:0 overruns:0 frame:0
TX packets:92809439 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1454141744 (1386.7 Mb) TX bytes:1454141744 (1386.7 Mb)



3) To find more information about a network interface, the below example shows detail information about interface eth0.


bash-2.05b# ethtool eth0







Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: Not reported
Advertised auto-negotiation: No
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Link detected: yes


File related commands
----------------------
1) find text in the matching files and show the filename with the pattern. The below example shows the list of filenames and the location of the hostname in those files, which will be helpful when changing hostnames for your websphere installation.

find . -name '*.xml' -print | xargs grep 'hostName' /dev/null







bash-2.05b# find /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/ -name 'server*.xml' -print | xargs grep 'hostName' /dev/null | more
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/portalserverNode01/serverindex.xml:<serverindex:
ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5
.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com" endPointRefs="NamedEndPoint_1130375480667 NamedE
ndPoint_1130375480668 NamedEndPoint_1130375480669 NamedEndPoint_1130375480670 NamedEndPoint_1130375480671 NamedEndPoint_113037548067
2 NamedEndPoint_1130375480673">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2034indchn2.example.com/serverindex.xml:<
serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserv
er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130386859873" hostName="n2034indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2033indchn2.example.com/serverindex.xml:<
serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserv
er/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1130361145826" hostName="n2033indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/portalserverCellManager01/serverindex.xml: <ser
verindex:ServerIndex xmi:id="ServerIndex_1" endPointRefs="NamedEndPoint_1 NamedEndPoint_2" hostName="portalserver.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2011indchn2Node01/serverindex.xml:<serverindex:
ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5
.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="n2011indchn2.example.com" endPointRefs="NamedEndPoint_1130375717654 NamedE
ndPoint_1130375717655 NamedEndPoint_1130375717656 NamedEndPoint_1130375717657 NamedEndPoint_1130375717658 NamedEndPoint_113037571765
9 NamedEndPoint_1130375717660">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2033indchn2UnNode01/serverindex.xml:<serverinde
x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas
/5.0/serverindex.xmi" xmi:id="ServerIndex_1173388344070"hostName="n2033indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/portalserverCell01/nodes/n2034indchn2UnNode01/serverindex.xml:<serverinde
x:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas
/5.0/serverindex.xmi" xmi:id="ServerIndex_1173388421900" hostName="n2034indchn2.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/templates/servertypes/APPLICATION_SERVER/serverindex.xml:<serverindex:ServerI
ndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/schemas/5.0/serv
erindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/cells/portalserverNode01Cell/nodes/portalserverNode01/serverindex.x
ml:<serverindex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/app
server/schemas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="portalserver.example.com">
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/config/backup/base/templates/servertypes/APPLICATION_SERVER/serverindex.xml:<serveri
ndex:ServerIndex xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:serverindex="http://www.ibm.com/websphere/appserver/sche
mas/5.0/serverindex.xmi" xmi:id="ServerIndex_1" hostName="$(node.host.name)">



2) Soemtimes you might run out of diskspace and want to clean up some files, so this command will help you find all large files. In this example it will list all the files greater than 10 mb in size.

find / -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5







bash-2.05b# find /usr/IBM/WebSphere -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5-rw-r--r--
1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/tranlog/log1
-rw-r--r-- 1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/partnerlog/log2
-rw-r--r-- 1 root root 1047553 Oct 26 2005 /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/tranlog/n2110indch2Cell01/n2110indch2Node01/n2110indch2_server1/transaction/partnerlog/log1


3) When sending SystemOut.log or SystemErr.log or trace.log to your developers or to IBM support Please use Tar and gzip on the fly to reduce the size of the file and the transfer time to FTP,email or http upload.

To pack on the fly...
tar -cvf - FILE-LIST | gzip -c > FILE.tar.gz

To unpack on the fly..
gunzip < FILE.tar.gz | tar -xvf -

To pack on the fly and email the file to IBM Support
tar -cvf - SystemOut.log | gzip -c | uuencode SystemOut.tar.gz | mail -s "PMR 09921,227,000" -b "SystemOut.log for server crash " WebSphere_Support@mainz.ibm.com

To find selective files and tar it up on the fly...
find /opt/IBM/WebSphere/PortalServer/log -name 'System*.log'| xargs tar -rvf System_logs.tar


4) Sometimes you might encounter a situation for example when running xmlaccess or wsadmin where you might want to write stdout to the file as well as to the console so that you have a copy even when the standard output disappears because of the screen buffer size.


ls -l | tee > filename
xmlaccess.sh -in ExportRelease.xml -username wpsadmin -password wpsadmin -url http://localhost:9080/wps/config | tee filename
./wsadmin.sh | tee wsadmin.log

5) find files that is recently updated which might be useful to see what changes are made to the system recently (e.g) below command shows the files that are modified within the last 20 minutes(-mmin) and 20 days (-mtime)


find /usr/IBM -mmin -20
find /usr/IBM -mtime -20








/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/dmgr/SystemOut.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/ffdc/dmgr_exception.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/activity.log
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer0.ser
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer1.ser
/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/wstemp/events/eventbuffer2.ser
/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs




6) Find a memory usage of the system.


free
cat /proc/meminfo








bash-2.05b# free
total used free shared buffers cached
Mem: 11819592 4938416 6881176 0 298148 1440200
-/+ buffers/cache: 3200068 8619524
Swap: 2048276 0 2048276

bash-2.05b# cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 12103262208 5056954368 7046307840 0 305303552 1474797568
Swap: 2097434624 0 2097434624
MemTotal: 11819592 kB
MemFree: 6881160 kB
MemShared: 0 kB
Buffers: 298148 kB
Cached: 1440232 kB
SwapCached: 0 kB
Active: 3718616 kB
ActiveAnon: 2968776 kB
ActiveCache: 749840 kB
Inact_dirty: 789328 kB
Inact_laundry: 237596 kB
Inact_clean: 0 kB
Inact_target: 949108 kB
HighTotal: 11140992 kB
HighFree: 6681628 kB
LowTotal: 678600 kB
LowFree: 199532 kB
SwapTotal: 2048276 kB
SwapFree: 2048276 kB
Committed_AS: 6039100 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB


4) To repeat a command contimuously.

watch -d=10 -n 1 "netstat -a"







Every 1s: netstat -a| grep ESTABLISHED Sun Feb 3 21:48:04 2008

tcp 0 0 n1212indchn2.www.com:9900 n1312indchn2.www.com:11530 ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9353 n1312indchn2.www.com:11549 ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9354 n1312indchn2.www.com:11550 ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9201 n1312indchn2.www.com:11544 ESTABLISHED
tcp 0 0 n1212indchn2.www.com:9352 n1312indchn2.www.com:11545 ESTABLISHED

2 comments:

Sobhan said...

Usefull stuff.

Gopi said...

Very Usefull Stuff.If you have few more daily used commands plz post.

There was an error in this gadget