Monday, March 24, 2014

Thread Dump Analysis

Dear WebLogic users/Administrator you might already visited many other sites that discussed about - how to take the thread dump. But here I am going to tell you about automation script makes more simplified way of working, with more information - almost 70% on the occasion of STUCK thread Thread dumps will reveals. In case of High CPU utilization situations, which method is badly working.

Step by Step analysis for Thread Dump

Step 1: Automated Thread Dump for WebLogic

Take the Thread dump for a managed server that have issues or slowness or latency etc. You just read the description and start applying to your test environment and then try out on your Live environments. Cheers!

# This script will takes the managed server name as argument and it will work generic to Linux or SunOS or HP-UX.
# The only ps command options varies for each platform
# Kill -3 command used for every 10 seconds of interval. You can increase the number of dumps in the for loop


   case $OSENV in
      Linux)WLSPID=`ps auxww|grep weblogic|grep -v grep|grep $WL_SERVER | awk '{print $2}'`;;
      SunOS)WLSPID=`/usr/ucb/ps -auxww|grep weblogi[c]|grep $WL_SERVER | awk '{print $2}'`;;
      HP-UX)WLSPID=`ps -efx|grep -v grep|grep weblogic|grep $WL_SERVER | awk '{print $2}'`;;
   if [ ! -z "$WLSPID" ]; then
      for i in 1 2 3 4 5
         echo "Generating thread dump number $i in logs/${WL_SERVER}_stdout.log"
         jstack  $WLSPID >> /log/${WL_SERVER}_stdout.log
         sleep 10
      echo "Error!!! process id not found for $WL_SERVER"

Step 2: Start analizing Thread dump

Take the sed sword into your hands !! sed UNIX utility will give you the right way of using regular expressions to expose desired outputs. The following is more effective script for you, love it share it :)
# Script Name:
# Description: Pass the threa dump log file name as argument to this script it will show you the required lines that need to see 
# Regular expression pattern is applied for search in the log file.

for i in `egrep 'trying to get lock|waiting to lock' $1 |  sed -n 's/.*[<@]\(.*\)[>[].*/\1/p' | sort | uniq`
 # Under that above line you will look thru the log for locked or holding lock
    sed -e '/./{H;$!d;}' -ne 'x;/ locked/{;/'${i}'/p;};/Holding lock/{;/'${i}'/p;}' $1
Look for more deeper with the handy utility that uses python. The following script can work to convert the given decimal number to hexadecimal number.
USAGE="Usage: $0 {number}"
if [ ! $# == 1 ]; then
  echo $USAGE
python -c "print hex($1)"
exit 0

The following script can work to convert the given decimal number to hexadecimal number.

If you want to generate a separate thread dump file with prstat, pstack commands output it will be much wise. Everything in single file to analysis at the time of critical situations. Ofcourse it happen only in UNIX environments :).

WebLogic Application performance problems: Diagnosing

The following presentation will take you to understand with a clarity on OOME types and what are all the reasons. It also address the High CPU issues.
Write a comment what do you thing on this issues. Happy trouble shooting!!!

RegEx for WebLogic Admin

Trick on grep

Normally every WebLogic user/Admin knows that "ps -ef|grep java" gives you the WebLogic servers process list in any UNIX or Linux environments. Most of the forums suggesting use "|grep -v" to exclude the "grep line". But, here I don't want to use one more pipe and grep -v option really makes expensive in CPU cycles. Searched for the optimistic solutions and finally found! Two choices using --exclude  and using regex.

The exclude options is not available on all UNIX systems. So I've chosen next option.

How do you avoid the grep line in the grep command output itself?
Here the trick is using the regular expression to workout this requirement. Use your searching pattern any one of the character in [] then it will automatically excludes the grep line from the output. actually it wouldn't get output itself!

bash-2.05$ ps -ef |grep jav[a]
weblogic   950   934  0   Feb 10 ?        0:06 /usr/local/bea/jdk142_11/bin/java -client -Xms32m -Xmx200m -Xverify

Note here I have tried out this trick on Sun Solaris 9 machine. Write your experiments in the comment box, share it!

Sunday, March 16, 2014

WebLogic Server name its PID, CPU, MEM

I am on a 'THINKING CHAIR' again, thinking to find the fine chopped shell script for displaying process id on the opposite to it the WebLogic(java process) instance name. This is the regular need for any WebLogic Administrator.

1. Some times the Thread counts monitoring cannot help to know the status of the Server instance then need to look for the instance is alive or not. That is the java process exist for that particular instance.
2. You may get Stuck Thread alert for any machine through your monitoring tools (HP OVO, Introscope or some other). Then you need to get the thread dump for the stuck thread found WebLogic instance.
3. You may want to kill an instance for any abnormal reasons

What is 'ps' command does?

The 'ps' command gives us information about processes status on Solaris or Linux or any other UNIX flavor.  There are different varieties of 'ps' command paths
i) /usr/bin/ps
ii) /usr/ucb/ps

If you type “ps” at the prompt, and you get very little information about the process.

ps command switches

The ps command switches are key, as always required to get the desired output of ps command. To get a complete list, for instance, type switches with prefix - symbol.

  • “ps -A” (Note: Linux is case-sensitive). Also, 
  • “ps -e” will give you a complete list of processes running on the machine. 
  • "ps -f" give a full listing, i.e. more information about each process listed 
  • "ps -u username" will list all processes running by a specific user mentioned as argument value
  • "ps  -l" gives a long listing of processes. This is even long data than in a full listing

The power to this 'ps' command will be filled up with its options, If you know how to use the options you can make wonders!! Now I need to know what are those options which could help me to write the script. You tell me what to do? Well your answer is go to our super duper search engine 'Google'.  ps(1B) man page on Oracle sun site. I selected -a, -ww, -x options that could help me to get complete command details of a process id associated with it.

Now I need to use 'cut' command for the WebLogic instance name from the big process detailed text. This is really challenge dear, what you suggest dear super reader? After running between many sites and man pages I got to know few ideas on awk and its family nawk command script.

This script is a generic script I had executed for WebLogic 9.2, 11g also. You can apply this for WebLgoic 7/8.1 also.

# FileName: 
# This script will Fetches PID of WebLogic Server instances 
# =========================================================
echo "PID associated with WebLogic instances"
echo  "====================================="
/usr/ucb/ps -awwx | grep "weblogic.Name" | grep -v "grep weblogic.Name" | nawk 'BEGIN {print "PID\tWLServer";
print  "=====================================" } ;
        NUM = match($0, "weblogic.Name=") ;
        START_STR = substr($0, START_POS) ;
        FINISH = match(START_STR, " ") ;
        FINISH_STR = substr($0, START_POS, FINISH_POS) ;
        NUM = split(FINISH_STR,FINISH_ARRAY) ;
        printf ("%s\t%s\n",$1, FINISH_ARRAY[1]) ;
        END {
        print "===================================="}'

#=== Published on  ===== 

The above script is executed on Solaris 9, 10 the output will look like as follows:
PID associated with WebLogic instances
PID     WLServer
1354    admin
11469   mserver01
11581   mserver02


Similar type of script is required for Linux machines too. Then I need to change the above script in 3 places.

The ps command options, Linux supports gawk (GNU awk) instead of nawk. The final thing to change is the string that requires not to grep useless string.

Perl for WebLogic on Linux
With the ps command we can get the desired output onto standard output. Perl having great Match, split built-in functions. In Linkedin Perl group, I followed Ashutosh Kukreti tips. You must be member so that you can see the discussion. I had posted the question about my requirement. After looking into Perl regular expressions usage.

$plist = `ps -U \$LOGNAME -o pid,pcpu,size,cmd|grep -i weblogic.Name|egrep -v grep`;
@prlist = split(/\n/ ,$plist );
print "PID    SERVER          CPU     SIZE      DOMAIN DIR
------ --------------- ------ ------  -----------------------------------
foreach $_ (@prlist){
        $x=~ m/^\s*(\d+)\s.+\s+-Dweblogic\.Name=(\S+)\s+/ ||next;
        my @values = split(' ',$x);
        if($cpu > $CPU_THRESHOLD)
                print "\n\nSUGGESTED ACTION:\n"
                        . "-----------------\n"
                        . "Use this command to trace threads on servers "
                        . "with over $CPU_THRESHOLD% CPU usage:\n"
                        . "/usr/bin/kill -3 " . $pid."\n";
        $r = `/usr/bin/pwdx $pid 2>&1`;
        $r =~ s/^\d+://;
        $r =~ s/^\s+|\s$//g;
        print $pid, "\t", $server,"\t",$cpu,"\t",$mem,"M\t",$r,"\n";

This part of Perl script, my friend SRIKANTH PANDA helped me to get process id and its related info with ps command.

weblogic automation - process details, memory details with utility script
WebLogic PID, CPU Load, MEM Automation script - Perl Script

Here in this script you can set your application desired CPU_THRESHOLD value to get notifying mails. After saving the file provide the execution permissions with chmod u+X The following is the output which I've executed on Ubuntu Linux with WebLogic 12c.

$ ./
PID    SERVER          CPU     SIZE      DOMAIN DIR
------ --------------- ------ ------  -----------------------------------
2518 AdminServer 2.8 860M /home/pavanwla/wlsdomains/medrec

Enjoy the fun with scripting tricks on your latest WebLogic environments, keep posting your updates and thoughts on this blog. Share this blog in your technical blogs.

Blurb about this blog

Blurb about this blog

Essential Middleware Administration takes in-depth look at the fundamental relationship between Middleware and Operating Environment such as Solaris or Linux, HP-UX. Scope of this blog is associated with beginner or an experienced Middleware Team members, Middleware developer, Middleware Architects, you will be able to apply any of these automation scripts which are takeaways, because they are generalized it is like ready to use. Most of the experimented scripts are implemented in production environments.
You have any ideas for Contributing to a Middleware Admin? mail to me