Why long way for thread dump on UNIX/Linux machines? Easy Steps!

jps - Java process list : Java Utilitycommand 

I was looking for diagnostic ways with freely available tools from JDK then I found the great valuable command tool 'jps'. The jps command can be used with three options(l, v, m). it can work for any Java - JEE Servers such as WebLogic, WebSphere, JBoss, Tomcat. In other words, any app server that uses the latest JDK 5+ version (version 6, 7, 8 also supports). Let me walk-through those interesting options in Win and *nix platforms. Before you execute this command tool, make sure that JAVA_HOME\bin is in the PATH setting.

Java Command tool : jps options

jps Option : jps -l

this -l option will give the full Java package name that invoked the Java process.

C:\Users\pavanbsd>jps -l
9116 sun.tools.jps.Jps
8812 weblogic.Server

Filtering the weblogic instance with find command on jps will give you the desired outcome.

C:\Users\pavanbsd>jps -l |find "weblogic"
8812 weblogic.Server

jps -v

this -v option is most important for tracking WebLogic, this option gives you the JVM aguments used to invoke the JAVA process (weblogic.Server). Usually for our WebLogic what we set in the setDomainEnv.sh will appear with this option.

C:\Users\pavanbsd>jps -v |find "weblogic"
8812 Server -Xms256m -Xmx512m -XX:MaxPermSize=128m -Dweblogic.Name=adm_clstr_dom -Djava.security.policy=C:\Oracle\MIDDLE
~1\WLSERV~1.3\server\lib\weblogic.policy -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=C:\Oracle\MIDDLE~1\WL
SERV~1.3 -Dwls.home=C:\Oracle\MIDDLE~1\WLSERV~1.3\server -Dweblogic.home=C:\Oracle\MIDDLE~1\WLSERV~1.3\server -Dweblogic
.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext
.dirs=C:\Oracle\MIDDLE~1\patch_wls1036\profiles\default\sysext_manifest_classpath

jps -m

this -m option will gives you the Java main() programs command line arguments that used by Java process.

 
C:\Users\pavanbsd>jps -m
8812 Server
8984 Jps -m

The problem of finding Java PIDs

Of course, you can kill the instance in the emergency in the same way this is quicker than regular ps command. Usually, WLA uses the following command for finding the WebLogic managed server process:

ps -fu username

Scanning that process list with your eyes for WebLogic start process, then its child process finally java process in that process list. the whole process looks hectic when there are many WebLogic instances running on the same machine. Sometimes killing wrong process makes nonsense/messy. To avoid such collisions in the decision of killing process best way is using 'jps'.

Try it and know about it then you can enjoy working with it...

Another problem area that will do pain for Middleware admins is that Find is the NodeManager is running or not on a machine (Win/*nix)? the best option I have simply types 'jps' it will give the desired result. Here I found the simple way for taking thread dump with JDK tools which work on the UNIX machine and also in Windows. Assuming that you WLA(WebLogic Admin) are aware of which WebLogic instance needs to take Thread-dump for analysis of an issue. The best option which I use that makes it very easy to do so is the java process in a UNIX box is :

$ jps -lv |grep WL_instancename

This will give you the process id and long view of JAVA_OPTIONS assigned to that instance. Pick that first number (Java PID) and use your regular kill -3 on it :)

The jps command as Troubleshooting helper

The jps command can be used to pipe out the PID for thread-dumps and memory/heap-dumps or Generation wise footprint with the following JDK commands utilities:

• jstat 
• jstack
• jhat 
• jdb
• jmap
• jinfo
• jvisualvm
• jconsole

Count WebLogic Servers using jps

Basically while working on Produciton environments, there would be WebLogic server bounce required while doing Application release or some patching happen on some of the middleware architecture components. Sometimes you may need to know how many WebLogic instances are running in that Unix machine or how many left for bounce. Simply you can use our free 'jps' command as shown below

 $jps |wc -l

If you wish to see only java process you can use simply jps command.

$jps

We can use for the WebLogic 9.x 10, 11g versions managed instances for thread dumps or terminating the process.

The same trick will work for all Oracle Fusion Middleware Infrastructure using services such as Oracle Analytics Server (OAS) domain, Oracle Data Integrator (ODI) domain, Oracle Business Intelligence Enterprise Edition (OBIEE), Oracle APEX domain, Oracle SOA Suite, Oracle Service Bus (OSB) domain, OUD, OIM and may more.

Note:
1. This command is available in JDK version 1.5 onwards only.
2. Careful while using long view it shows full details.

Script for Bouncing a WebLogic instance

In the last post I was made deeper research on getting the process id of all WebLogic instances with a script. After looking to that script, My asked me 'Why don't you try for a script for bouncing a WebLogic instance?'. I thought that's really good idea, this makes WebLogic Admin life more easier, which is very much need for every WebLogic Admin. Let me get all the clues before putting the logic into the script.






The bounce script will take the input as WebLogic instance name.
Then the script should do search for the corresponding Java Process id of given WebLogic instance.
That Java Process ID can be used for finding:
1. Thread dump
2. Domain directory
3. Shutdown instance (kill the process)


Above 1, 3 are commonly used commands but 2nd one required when situation like this, on the same machine if there exists multiple WebLogic domain instances then how to find the WebLogic instance started from? Here, I need to find the origin directory of startManagedWebLogic.sh used which gives the WebLogic domain directory. How to resolve this? The solution is pwdx command in Solaris gives you this directory path by passing process id to it.

This solution gives me to start the WebLogic instance which is residing in irrespective WebLogic domain. That's all I need to make the bounce script.

The bounce script goes like this

#==========================================================
# This script you can keep in common accessing PATH
# such as in $JAVA_HOME/bin
# For WebLogic instance take ThreadDump, Shutdown, and Start
#===========================================================
flag=0
if [ "$1" = "" ]; then
        echo "Usage : $0 [instance]"
        exit
else
        instance="$1"
        pid=`jps -v|grep $instance|awk '{print $1}'`
        echo 'pid ' $pid
        domainPath=` pwdx $pid |awk '{print $2}'`
 
        kill -3 $pid
        sleep 3
        kill -3 $pid
        sleep 3
        kill -3 $pid

 #Now shutting down the instance here
        kill -9 $pid

 #verifying the instance is existance if not wait
        while : 
        do 
                jps -v |grep $instance
                if [ $? -eq 1 ]; then
                        echo 'kill success'
                        flag=`expr $flag + 1`
                        break;
                else 
                        echo 'kill failed... sleeping 1 sec'
                        sleep 1
                fi 
        done 

 #Once kill is success then you can proceed for Start it
 # Here you can call startManagedWebLogic.sh script
        if [ $flag -gt 0 ]; then
                . ${domainPath}/startinstance.sh $instance
        fi
fi

Recently one of my blog ->pingbox user asked me about pwdx command, does it works on HP-UX? I thought I should update this post with proper references.
There is compatibility list given for pwdx command:
pwdx compatibility

Alternate solution is cwd for Linux.
Please suggest your ideas, Keep commenting

JVM monitoring with jstat Shell Script

The Oracle WebLogic Server instances are nothing but each one running on a single JVM, when this JVM crashes there could be one of the reasons as overworking of Garbage Collection or not working at all (waiting continuously).  

It is good practice that monitoring GC in JVM with detailed statistics will give you a clear idea to analysis at what circumstances happening wrong. And the best way to look in deeper levels of garbage collection also like Young Generation (Eden space, Survivor spaces S0, S1) Old Generation (tenured Generation), and Perm Generation (Statistic Objects/Classes).

JDK 1.5 and latest providing excellent JDK command utilities for interrogate the current running Java Process and look inside of JVM take snap with following:
1. jps (Java Process)
2. jstat (JVM status)

The 'jps' command with -lv options gives you complete detailed java process arguments for MEM settings and relavent WebLogic Server instances name. We have already discussed about this command utility in another post.

The jstat command with -gc option will produce the Garbage Collection of given java process id.

GC Monitoring shell script with jstat tool

I have modified according to my convenience and requirements as a shell script that will take the argument as WebLogic Server instance name and give us the output as statistics of Young, Old, Perm Generations Current, Used Memory details in MB conversion function.

##############################################################
# Purpose:  To display current, used spaces for an instance 
#   in the JVM Eden, Old, Perm sizes
# Author:  Pavan Devarakonda
##############################################################
# Function for converting the Kilo bytes to Mega Bytes.
toMb()
{
        echo 'scale=0;'$1'/1024' | bc; 
} 

#=============================================
# JVM status Function starts here
#=============================================
instJvm()
{
        DATA=`$JAVA_HOME/bin/jstat -gc $PID | grep -v S0C`
         
        EC=`echo $DATA | awk '{print $5}'`
        EU=`echo $DATA | awk '{print $6}'`
        OC=`echo $DATA | awk '{print $7}'`
        OU=`echo $DATA | awk '{print $8}'`
        PC=`echo $DATA | awk '{print $9}'`
        PU=`echo $DATA | awk '{print $10}'`
        echo -e  "$2\t|" `toMb $EC`"M   "`toMb $EU`"M\t| "`toMb $OC`"M   "`toMb $OU`"M\t| "`toMb $PC`"M   "`toMb $PU`"M"
}
#=============================================
# main starts here
#=============================================
 
 
echo -e "=============================================================="
echo -e " Instance\t |    Eden    \t|    Old Gen   \t|    Perm Gen"
echo -e "         \t |Current Used\t| Current Used \t| Current Used"
echo -e "=============================================================="
 
DOMAIN_HOME=$1

for i in ` ls $DOMAIN_HOME/servers/|grep -v bkp`
do 
        PID=`$JAVA_HOME/bin/jps -lv | grep $i | awk '{print $1}'`
        if [ "$PID" != ""  ]
        then    
                instJvm $PID $i
        fi      
done
echo -e "=============================================================="

Example execution:

  ./jstat_mon.sh  /u01/domains/apex_domain
  

The output as follows:

The jstat command using script

        

Taking Advantage of Naming Conventions

In most of the Middleware environments, the WebLogic domain will have admin servers, managed servers use naming convention construct that will have some common identity for each domain unique. In the above example, The domain uses naming convention with the letter 'c', This can be any word or letter that is used in your environment. To make your script you need to replace 'c' with your enviroment key word/letter. (Thanks Raman Jain for your valuable comment)

Let me show you the sample Output:

Looking ahead to your valuable comments and suggestions to improve our scripting tricks for better WebLogic Administration.

The jstat Command reference:

http://roundoverlinux.blogspot.in/2013/03/jstat-command-is-java-virtual-machine.html