Thread Dump Analysis

Dear WebLogic users/Administrator you might already visited many other sites that discussed about - how to take the thread dump. But here I am going to tell you about automation script makes more simplified way of working, with more information - almost 70% on the occasion of STUCK thread Thread dumps will reveals. In case of High CPU utilization situations, which method is badly working.

Step by Step analysis for Thread Dump

Thread dump troubleshooting from Jerry Chan

Step 1: Automated Thread Dump for WebLogic

Take the Thread dump for a managed server that have issues or slowness or latency etc. You just read the description and start applying to your test environment and then try out on your Live environments. Cheers!

#!/bin/bash

# This script will takes the managed server name as argument and it will work generic to Linux or SunOS or HP-UX.
# The only ps command options varies for each platform
# Kill -3 command used for every 10 seconds of interval. You can increase the number of dumps in the for loop

WL_SERVER=$1
OSENV=`uname`

   case $OSENV in
      Linux)WLSPID=`ps auxww|grep weblogic|grep -v grep|grep $WL_SERVER | awk '{print $2}'`;;
      SunOS)WLSPID=`/usr/ucb/ps -auxww|grep weblogi[c]|grep $WL_SERVER | awk '{print $2}'`;;
      HP-UX)WLSPID=`ps -efx|grep -v grep|grep weblogic|grep $WL_SERVER | awk '{print $2}'`;;
   esac
   if [ ! -z "$WLSPID" ]; then
      for i in 1 2 3 4 5
      do
         echo "Generating thread dump number $i in logs/${WL_SERVER}_stdout.log"
         jstack  $WLSPID >> /log/${WL_SERVER}_stdout.log
         sleep 10
      done
   else
      echo "Error!!! process id not found for $WL_SERVER"
   fi

Step 2: Start analizing Thread dump

Take the sed sword into your hands !! sed UNIX utility will give you the right way of using regular expressions to expose desired outputs. The following is more effective script for you, love it share it :)

# Script Name: ThreadAnalizer.sh
# Description: Pass the threa dump log file name as argument to this script it will show you the required lines that need to see 
# Regular expression pattern is applied for search in the log file.
clear

for i in `egrep 'trying to get lock|waiting to lock' $1 |  sed -n 's/.*[<@]\(.*\)[>[].*/\1/p' | sort | uniq`
do
 # Under that above line you will look thru the log for locked or holding lock
    sed -e '/./{H;$!d;}' -ne 'x;/ locked/{;/'${i}'/p;};/Holding lock/{;/'${i}'/p;}' $1
done

Look for more deeper with the handy utility that uses python. The following script can work to convert the given decimal number to hexadecimal number.

USAGE="Usage: $0 {number}"
if [ ! $# == 1 ]; then
  echo $USAGE
  exit
fi
 
python -c "print hex($1)"
exit 0

The following script can work to convert the given decimal number to hexadecimal number.

If you want to generate a separate thread dump file with prstat, pstack commands output it will be much wise. Everything in single file to analysis at the time of critical situations. Ofcourse it happen only in UNIX environments :).

WebLogic Application performance problems: Diagnosing

The following presentation will take you to understand with a clarity on OOME types and what are all the reasons. It also address the High CPU issues.

Java Application Problems: Diagnostics from Віталій Тимчишин

Write a comment what do you thing on this issues. Happy trouble shooting!!!