Saturday, April 5, 2014

Check system information for Capacity Planning

Working for migration projects requires lots of mesurements about the current running environment that gives you right way for predictions, estimations about the newly build environment. Thinking BIG!!! as architect for the enterprise applications, I started digging details of the system information of every machine I mean, every box and I need to prepare a table that will give all the details about the machine. This will make easy for deciding various things for WebLogic domain and its related environment.

Script Objective: Capacity Measurements

The Capacity plan is critical where a Senior WebLogic Administrator or Infrastructure Architect need to be more smarter to make wise decisions on allocated infrastructure. If you know the mesurements you can tailor your environment to give best predicated application environment platform. I had seen there is a legacy(old) shell script which gives the required information for mesurements! But it is not really robust and effective to give the desired outputs. Some of the modern CPU architectures changed and the processor information given expanded way. I thought better to renovate it in myway.  What do you say? Yes! right hmmm, I on top of it :).

The script must give outcome following factors
1. CPU Architecture
2. Num CPUs and  CPU Speed
3. Operating System Release version
4. Memory Sized (RAM)
5. Server platform
6. Virtual IP Addresses

First understood each line it was written in the script, then I refined those lines as per my thoughts and tried to get the effective output. bugging my friends if I'm not able to get desired outputs.

One Generic factor is that, I want to make this script must executed on both Solaris and also on the Redhat Linux operating environment. And finally reached the Victory!

I wish you could execute and share your experiances on HP-UX, AIX platforms as well that could help many other scripting fans.
-->

Optimizing Script: Less pipes

Recently April 2014, While teaching UNIX & Linux Scripting for WebLogic Administrators revisited this script. The scripting can be more optimized when you use less pipes (|). The usage of two commands 'cat' and 'grep' on a text file here it is /proc/cpuinfo more costly why because the number of CPU cycle consumption is the factor, that why it is changed by Swetha here. The initial script used by Swetha on Ubuntu 12 got some of the info blank and non related outputs. Infact, Linux /proc/meminfo file provides you the RAM size in kb that is little incoveniance for us. Hence changed it to MB format also in this update.
Hope you are excited to see what is that script ? Here it is...

#!/bin/bash
# FileName: CheckSystem.sh
# Version            : 1.2
# Initial Drafted    : 2012/10
# Programmed         : Pavan Devarakonda
# Updated            : Swetha Reddy
# Description         : The bash utility script, display CPU, no of processors, Kernal version, RAM, Virtual IP, DNS
#
#===================================================================================================================
PATH=$PATH:/usr/ccs/bin:/usr/share/lib:/sbin:/usr/sbin:/usr/local/bin:/bin
export PATH

os=`uname`
case $os in
SunOS)
 nproc=`psrinfo |wc -l` ;
 prspeed=` psrinfo -v | grep -i Hz |awk '{print $6" " $7}'|uniq  `
 inet_addrs=`ifconfig -a |grep -i inet | awk '{print $2}' `
 mem=`prtconf |grep -i size`
 ;;
Linux)
 nproc=` grep -i processor /proc/cpuinfo |wc -l`
 prspeed=` grep -i model /proc/cpuinfo |grep -i hz |awk '{print $9 }' `
 inet_addrs=`ifconfig -a |grep -i inet | awk '{print $2}' |cut -d ":" -f 2`
mem=` grep -i MemTotal /proc/meminfo | awk '{print $2}' `
mem=$(($mem/1024))
 ;;
esac
hname=`hostname`

echo "Host name : $hname"
echo "Operating system is $os release `uname -r `"
echo "Number of CPU : $nproc with the speed of $prspeed"
echo "RAM Size: " $mem"mb"

for IPAddr in $inet_addrs
do
 if [ $IPAddr != "127.0.0.1" ]
 then
  if [ $os = "Linux" ]
  then

   IPAddr_name=`nslookup $IPAddr|grep -i name |awk '{print $4}'`
  else
   IPAddr_name=`nslookup $IPAddr|grep -i name |awk '{print $4}'`
  fi
  if [ ! -z ${IPAddr_name} ]; then
   echo $IPAddr === $IPAddr_name
  fi

 fi
done


-->


After preparing my statistical table, reviewed and got an idea for include the disk space avaialble on every machine. That is the great result of good reviewer (my buddy did it!). So I need a disk size that I can get by df command but -lk is going to give in kilobytes, option -h works on Linux, Solaris 10 but summing up is issue so I took kilobytes and did convertions as follows:
df -lk | egrep -v "Filesystem|/proc|/dev/fd|swap|/home|/platform" | awk '
 { t += $2 }
 { u += $3}
 {GB2 = 1024*1024} END 
 { printf "%d of %d Gb in use.\n", u/GB2, t/GB2 }'
This command line excludes header, /proc, floppy, swap, user space and sun packages in /platform for Veritas.

35 of 130 Gb in use
Cheers! enjoy have fun in your work!
Good Reference:

http://www.brandonhutchinson.com/Gathering_Solaris_system_statistics.html

Keep writing your innovative ideas, generic suggestions in comments!!

Monday, March 24, 2014

Thread Dump Analysis

Dear WebLogic users/Administrator you might know from several other sites that, how to take the thread dump. But here I am going to show you how a script makes more simplify your work, with more information on the occasion of STUCK thread. High CPU utilization situations.

Step by Step analysis for Thread Dump


Step 1: Automated Thread Dump for WebLogic

Take the Thread dump for a managed server that have issues or slowness or latency etc. You just read the description and start applying to your test environment and then try out on your Live environments. Cheers!
#!/bin/bash

# This script will takes the managed server name as argument and it will work generic to Linux or SunOS or HP-UX.
# The only ps command options varies for each platform
# Kill -3 command used for every 10 seconds of interval. You can increase the number of dumps in the for loop

WL_SERVER=$1
OSENV=`uname`

   case $OSENV in
      Linux)WLSPID=`ps auxww|grep weblogic|grep -v grep|grep $WL_SERVER | awk '{print $2}'`;;
      SunOS)WLSPID=`/usr/ucb/ps -auxww|grep weblogi[c]|grep $WL_SERVER | awk '{print $2}'`;;
      HP-UX)WLSPID=`ps -efx|grep -v grep|grep weblogic|grep $WL_SERVER | awk '{print $2}'`;;
   esac
   if [ ! -z "$WLSPID" ]; then
      for i in 1 2 3 4 5
      do
         echo "Generating thread dump number $i in logs/${WL_SERVER}_stdout.log"
         kill -3 $WLSPID
         sleep 10
      done
   else
      echo "Error!!! process id not found for $WL_SERVER"
   fi

Step 2: Start analizing Thread dump

Take the sed sword into your hands !! sed UNIX utility will give you the right way of using regular expressions to expose desired outputs. The following is more effective script for you, love it share it :)

# Script Name: ThreadAnalizer.sh
# Description: Pass the threa dump log file name as argument to this script it will show you the required lines that need to see 
# Regular expression pattern is applied for search in the log file.
clear

for i in `egrep 'trying to get lock|waiting to lock' $1 |  sed -n 's/.*[<@]\(.*\)[>[].*/\1/p' | sort | uniq`
do
 # Under that above line you will look thru the log for locked or holding lock
    sed -e '/./{H;$!d;}' -ne 'x;/ locked/{;/'${i}'/p;};/Holding lock/{;/'${i}'/p;}' $1
done
Look for more deeper with the handy utility that uses python. The following script can work to convert the given decimal number to hexadecimal number.
USAGE="Usage: $0 {number}"
if [ ! $# == 1 ]; then
  echo $USAGE
  exit
fi
 
python -c "print hex($1)"
exit 0

The following script can work to convert the given decimal number to hexadecimal number.

If you want to generate a separate thread dump file with prstat, pstack commands output it will be much wise. Everything in single file to analysis at the time of critical situations. Ofcourse it happen only in UNIX environments :).

WebLogic Application performance problems: Diagnosing

The following presentation will take you to understand with a clarity on OOME types and what are all the reasons. It also address the High CPU issues.
Write a comment what do you thing on this issues. Happy trouble shooting!!!

Blurb about this blog

Blurb about this blog

Essential Middleware Administration takes in-depth look at the fundamental relationship between Middleware and Operating Environment such as Solaris or Linux, HP-UX. Scope of this blog is associated with beginner or an experienced Middleware Team members, Middleware developer, Middleware Architects, you will be able to apply any of these automation scripts which are takeaways, because they are generalized it is like ready to use. Most of the experimented scripts are implemented in production environments.
You have any ideas for Contributing to a Middleware Admin? mail to me wlatechtrainer@gmail.com
QK7QN6U9ZST6