Showing posts with label bash. Show all posts
Showing posts with label bash. Show all posts

Saturday, April 5, 2014

Check system information for Capacity Planning

Working for migration projects requires lots of mesurements about the current running environment that gives you right way for predictions, estimations about the newly build environment. Thinking BIG!!! as architect for the enterprise applications, I started digging details of the system information of every machine I mean, every box and I need to prepare a table that will give all the details about the machine. This will make easy for deciding various things for WebLogic domain and its related environment.

Script Objective: Capacity Measurements

The Capacity plan is critical where a Senior WebLogic Administrator or Infrastructure Architect need to be more smarter to make wise decisions on allocated infrastructure. If you know the mesurements you can tailor your environment to give best predicated application environment platform. I had seen there is a legacy(old) shell script which gives the required information for mesurements! But it is not really robust and effective to give the desired outputs. Some of the modern CPU architectures changed and the processor information given expanded way. I thought better to renovate it in myway.  What do you say? Yes! right hmmm, I on top of it :).

The script must give outcome following factors
1. CPU Architecture
2. Num CPUs and  CPU Speed
3. Operating System Release version
4. Memory Sized (RAM)
5. Server platform
6. Virtual IP Addresses

First understood each line it was written in the script, then I refined those lines as per my thoughts and tried to get the effective output. bugging my friends if I'm not able to get desired outputs.

One Generic factor is that, I want to make this script must executed on both Solaris and also on the Redhat Linux operating environment. And finally reached the Victory!

I wish you could execute and share your experiances on HP-UX, AIX platforms as well that could help many other scripting fans.
-->

Optimizing Script: Less pipes

Recently April 2014, While teaching UNIX & Linux Scripting for WebLogic Administrators revisited this script. The scripting can be more optimized when you use less pipes (|). The usage of two commands 'cat' and 'grep' on a text file here it is /proc/cpuinfo more costly why because the number of CPU cycle consumption is the factor, that why it is changed by Swetha here. The initial script used by Swetha on Ubuntu 12 got some of the info blank and non related outputs. Infact, Linux /proc/meminfo file provides you the RAM size in kb that is little incoveniance for us. Hence changed it to MB format also in this update.

Formatted output in bash script

This time (Oct 2014) we are working on transition project and need to know about the system, Mr. Nageswaraiah Nanisetty helped me to update this script again for formatting the output in a nice format using printf and C style format definition. Tested this script in the VM, test environment as well working excellent. That is really nice to copy into excel directly by colon as delimiter.
Hope you are excited to see what is that script ? Here it is...

#!/bin/bash
clear
PATH=$PATH:/usr/ccs/bin:/usr/share/lib:/sbin:/usr/sbin:/usr/local/bin:/bin
export PATH
echo "Today's date is `date`, this is week `date +.%V.`"

format="\n%20s : %-50s"
os=`uname`
case $os in
SunOS)
 nproc=`psrinfo |wc -l` ;
 prspeed=` psrinfo -v | grep -i Hz |awk '{print $6" " $7}'|uniq  `
 inet_addrs=`ifconfig -a |grep -i inet | awk '{print $2}' `
 mem=`prtconf |grep -i size`
 ;;
Linux)
 nproc=` grep -i processor /proc/cpuinfo |wc -l`
 prspeed=` grep -i model /proc/cpuinfo |grep -i hz |awk '{print $9 }'|uniq `
 inet_addrs=`ifconfig -a |grep -i inet | awk '{print $2}' |cut -d ":" -f 2`
mem=` grep -i MemTotal /proc/meminfo | awk '{print $2}' `
mem="$(($mem/1024))MB"
    VENDOR=`cat /proc/cpuinfo | grep 'vendor_id' | cut -d: -f2 | awk 'NR==1'`
    MODEL=`cat /proc/cpuinfo | grep 'model name' | cut -d: -f2 | awk 'NR==1'|sed 's/ //g'`
    MHZ=`more /proc/cpuinfo | grep 'cpu MHz' | cut -d: -f2 | awk 'NR==1'`
   FLAVOR=`lsb_release -i |awk '{print $3}'`


esac
hname=`hostname`
arch=`arch`
uname=`uname -r`
printf "$format"  "Processor Arch" $arch
printf "$format"  "Host name" $hname
printf "$format"  "Operating system" $os
printf "$format"  "Release" $uname
printf "$format"  "Number of CPU" $nproc
printf "$format"  "CPU speed" $prspeed
printf "$format"  "RAM Size" $mem
printf "$format" "CPU Model" $MODEL
printf "$format" "Vendor" $VENDOR
printf "$format" "Flavor" $FLAVOR


echo
for IPAddr in $inet_addrs
do
 if [ $IPAddr != "127.0.0.1" ]
 then
  if [ $os = "Linux" ]
  then
   IPAddr_name=`nslookup $IPAddr|grep -i name |awk '{print $4}'`
  else
   IPAddr_name=`nslookup $IPAddr|grep -i name |awk '{print $4}'`
  fi
  if [ ! -z ${IPAddr_name} ]; then
        printf "$format" "IPaddress-DNS"  $IPAddr-$IPAddr_name
  fi

 fi
done





Execution of the script gives output as follows:
$ ./sysInfo
Today's date is Fri Oct  3 21:26:09 SAST 2014, this is week .40.

      Processor Arch : x86_64
           Host name : myrhel.vybhava.edu
    Operating system : Linux
             Release : 2.6.18-274.el5
       Number of CPU : 24
           CPU speed : 2.67GHz
            RAM Size : 54358MB
           CPU Model : Intel(R)Xeon(R)CPUX5650@2.67GHz
              Vendor : GenuineIntel
              Flavor : RedHatEnterpriseServer

       IPaddress-DNS : 192.168.1.105-myrhel.vybhava.edu


After preparing my statistical table, reviewed and got an idea for include the disk space avaialble on every machine. That is the great result of good reviewer (my buddy did it!). So I need a disk size that I can get by df command but -lk is going to give in kilobytes, option -h works on Linux, Solaris 10 but summing up is issue so I took kilobytes and did convertions as follows:
df -lk | egrep -v "Filesystem|/proc|/dev/fd|swap|/home|/platform" | awk '
 { t += $2 }
 { u += $3}
 {GB2 = 1024*1024} END 
 { printf "%d of %d Gb in use.\n", u/GB2, t/GB2 }'
This command line excludes header, /proc, floppy, swap, user space and sun packages in /platform for Veritas.

35 of 130 Gb in use
Cheers! enjoy have fun in your work!

Good Reference:

http://www.brandonhutchinson.com/Gathering_Solaris_system_statistics.html

Keep writing your innovative ideas, generic suggestions in comments!!

Wednesday, January 1, 2014

Setting up a best login profile on UNIX


Usually when you are working on building the platform for any application, you might be given UNIX (Solaris/Linux/HP-UX/AIX) boxes and new generation with Virtual boxes. which contains nothing you need to do many installations and configurations. To make your administrative task easy and simple we need to setup a fine tuned profile file. This profile file will be holding the multiple environment variables those are reusable when you log-on to UNIX Kernel  Here you can define shell functions or you can define simple single lined aliases that makes lengthy commands set to simple trimmed single letter or meaningful words.

What is user SHELL and  its profile association?


In your UNIX Kernel when a new user login will check the /etc/passwd file which shell is assigned by the super-user. It will look for the system profile and assign the environment values, then on top of it user profile can be loaded. Hence the variable is defined in system profile can be overridden by user profile. Example PATH environment variable that can be over written by user defined profiles most usual thing it is. So careful while defining such variables. It could behave differently for Solaris same line cannot work on Linux or on AIX box.

Why do we need this to be customize User profile?


Any Middleware admin may involve install and configuring the Apache or Oracle WebCenter Sites or spaces or SOA, Hadoop or Build and deployment platform may requires ANT or Gradle, Jenkins or Hudson setups. Every component that interacts with Middleware runs on the same UNIX Kernel need customized variables that can be referred from any directory or path, so we need them as global variables, this you can acquire by defining them as environment variables in SHELL profile. UNIX Operating system provides all shells associated with a  specific default system provided and a user profile script and that is in the hidden file which prefixed with dot(.) so it looks like .profile in your home directory. You can view them by ls -la command.

Where can I find this profile script?


Usually when you login to UNIX Kernel, you will be landing to a directory that is designated to your user HOME directory. Suppose you already working on some other path, you can simply use cd  or cd ~ and press enter the OS will takes to home directory. Alternatively you can use system defined environment variable by giving cd $HOME.



How do I know I am in which SHELL?


First you need to know about your user login SHELL(default). Every user can have different choice of having a shell. Once you have assigned user or role for WebLogic environment setup then  how do you know what SHELL it is? use the following:

$echo $SHELL
/bin/sh

How many SHELL do my env have?
To know this you could do little experiment on your prompt with the following commands

$ type bash
/usr/bin/bash


Then you can look in that directory all shell names ends with 'sh', list them with small regular expression trick on your ls command. The main trick here is using 'anchor' symbols, the $ symbol indicates ending of the word pattern, and you can use for avoiding the .sh files in that directory character classes can be negated with the [^] syntax.
[WLA@server~]$ cat /etc/shells
/bin/sh
/bin/bash
/sbin/nologin
/bin/tcsh
/bin/csh
/bin/ksh
...

[WLA@server~]$ ls /usr/bin | grep [^.]sh$
bash
csh
hash
jsh
ksh
pbksh
pbsh
pfcsh
pfksh
pfsh
remsh
rksh
rsh
ssh
tcsh
zsh
[WLA@server~]$ ls /usr/bin |grep [^.]sh$ |wc -l
cat /etc/shells

How to set bash as login SHELL?

I feel the Bourne Again Shell (bash) is best suitable login shell for WebLogic environment setup.
Sample .bashrc script which will be executed automatically when you hit 'bash' command at your Linux command prompt.
export JAVA_HOME=/u01/app/oracle/jdk1.8.0_121
export PATH=$JAVA_HOME/bin:$PATH
export MW_HOME=/u01/app/oracle/fmw122120
export WL_HOME=$MW_HOME/wlserver
alias wlst="$MW_HOME/oracle_common/common/bin/wlst.sh"

echo ===================================

Now your immediate question might be 'How do you know which shell is assigned to your user profile?' Here is the Trick for you!!

$ echo $SHELL
/bin/sh

The login shell will be assigned as per the common profile set by Solaris/Linux Administrator. That can be changed by passwd -e or passwd -r if you must have root access then only you can do editing the passwd file and update your desired shell there. Some times it is rejects you and shouts 'permission denied'. Just smile on it you can proceed with the following trick!! Using exec command, that will replaces the existing shell with new shell that you want.

export SHELL=/usr/bin/bash
exec $SHELL -l

Shell replacement ksh to bash with exec

What is a process?

process is an entity in execution. The process can be created by fork, exec system calls. the exec system call replaces current process. In the above also happens same for login shell. Example  If your default login shell is given as KSH(/bin/ksh), but in your .profile given exec command to replaces the shell with BASH(/bin/bash).


So, now I am ready with sample .bash_profile for WebLogic 11g for Solaris 10

Implementing the best practices all together, to make this most generic profile. 

#==============================================
# Sample .bash_profile or .bashrc or .profile for ksh
# Middleware Admin  on Solaris 10 operating environment
# Updated : October 8, 2010
#==============================================
clear

USERNAME=$LOGNAME
HOSTNAME=`hostname`
#---------- echo Welcome message ---------------------
echo
echo Welcome $USERNAME. You are on Server: $HOSTNAME !!!
export JAVA_HOME=$HOME/Oracle/Middleware/jdk160_18
#---------------WebLogic Specific ---------------------------------------
export MW_HOME=$HOME/Oracle/Middleware/
export WL_HOME=$MW_HOME/wlserver_10.3

export ORACLE_HOME=/opt/oracle/product/9.2.0
export LD_LIBRARY_PATH=$WL_HOME/server/native/solaris/sparc:$WL_HOME/server/lib/solaris/oci920_8:$ORACLE_HOME/lib32:$LD_LIBRARY_PATH
 
export THREADS_FLAG=native
export PATH=$PATH:.:$JAVA_HOME/bin:/usr/bin:/usr/ucb:/etc:$WL_HOME/bin:$ORACLE_HOME/bin:/usr/ccs/bin:/usr/share/lib:/sbin:/usr/sbin:/usr/local/bin:$WL_HOME/server/bin

export CLASSPATH=$WL_HOME/server/lib/weblogic.jar:$JAVA_HOME/jre/lib/rt.jar
#---------------- End of WebLogic specific ---------------------------------
export MANPATH=$MANPATH:/usr/share/man
export EDITOR=vi
export PS1="[\u@\h \w]\$ "
 
alias cls="clear"
alias sl=ls
alias cd..="cd .."
alias cd...="cd ../.."
alias gi="grep -i"
alias l="ls -al|grep '^d'"
alias lm="ls -al | more"
alias h=history
alias hm="history | more"
alias nu="who|wc -l"                # nu - number of users
alias np="ps -ef|wc -l"             # np - number of processes running
alias p="ps -ef"
alias help=man
alias path="echo $PATH"
##### End of the sample .bash_profile for smart WLA #################
#===== Published on http://wlatricksntips.blogspot.com =============#

Importance of DOT(.) in PATH
I just started using the above .bash_profile but, something missing in it... when I wish to start my WebLogic server from domain home I need to give dot and forward slash (./). This looks somewhat unusual, when I add a DOT( .) in PATH variable made everything normal and I can start my WebLogic instances with my customized script from the prompt.

Power of PATH
More power to your user shell with adding /usr/sbin most of the network, system relavent commands available to you at command prompt.

The changes I made for the PATH line adding JAVA_HOME/bin your latest Java binaries will be useful at command prompt such as jps, jstat, jmap very useful when you deal with heap dumps and memory issues, $WL_HOME/server/bin consists of most of the WebLogic related configuration and maintenance commands available to you, it is little trick you need adding this in Solaris and Linux because it uses L-R precedence on Solaris for Linux R-L use appropriately.

export PATH=$PATH:.:$JAVA_HOME/bin:/usr/bin:/usr/ucb:/etc:$WL_HOME/bin:$ORACLE_HOME/bin:/usr/ccs/bin:/usr/share/lib:/sbin:/usr/sbin:/usr/local/bin:$WL_HOME/server/bin

Java want CLASSPATH
When there is a need of running ANT or build scripts you need runtime that is rt.jar and tools.jar in CLASSPATH this very often.

If you are using JRockit instead of SunJava then you don't need the rt.jar or tools.jar files in CLASSPATH. Other than this we have WebLogic specific utility commands that can be run from any where if you add weblogic.jar into the CLASSPATH. You can run the following command line utilities:

  1. java weblogic.WLST
  2. java weblogic.Admin
  3. java weblogic.Deployer



Overcome Human Errors with Alias
Mistakes can be made by humans it is obvious thing. Mostly while working on UNIX environments while typing the command there could be mistype happens. Example very effective WebLogic troubleshooter in the pressure situations might mistype 'sl -l' instead of 'ls -l'. sometimes 'l s-lrt' these mistakes can be over come with proper identifying our typing styles :) define them as alias with correct command.

That looks pretty and easy to use. If you need any assistance write a comment or reach me on G+ or FB please don't shy!!  : )

Usage of umask

When your WebLogic server generating logs need common access to the other users who are just monitoring the system or auditing the system. Such users are not WebLogic admin user, even though they need to access your logs. For this privileges every WebLogic server logs folder you need to update using chmod command repeatedly. To resolve this best option is using umask.

Don't forget to write back your comments and queries.

Ubuntu profile setup in VM

Now a days Ubuntu became almost alternative for Windows OS with great UI features in a Linux flavors. In Ubuntu the default user shell is 'bash'. The order it will execute according to the system profile. The system contains bashrc. So you can place your environment variables int the .bashrc or else alternatively you can call the . ~/.bash_profile in the .bashrc
export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export PATH=$PATH:$JAVA_HOME/bin
export WL_HOME=/home/pavanwla/Oracle/Middleware/Oracle_Home/wlserver
export CLASSPATH=$WL_HOME/server/lib/weblogic.jar:.

Please share your experiance with profile and without knowing about the profile. Lets have best profile for your WebLogic environments that you build. References
1. http://www.unixguide.net/unix/bash/A7.shtml
2. http://djcraven5.blogspot.com/2008/03/change-solaris-shell-to-bash.html
3. http://www.cyberciti.biz/faq/solaris-unix-change-default-shell/
4. http://www.ibm.com/developerworks/aix/library/au-speakingunix_commandline/?S_CMP=cn-a-aix&S_TACT=105AGX52
5. The umask command usage

Tuesday, June 5, 2012

Check URL using shell script


We had an issue on production environment. Suddenly one of the webserver got crashed that no one aware of it. That was happen in the Weekend there were no users accessing at that time. When the business hour started Monday morning customer reported to service desk that one of the URL is not working for them. We middleware support team check the URL and identified that the HTTP server not working. Next action items as you know taking the logs backup and starting the HTTP server done.
As part of issue management preventive action to this there could be a monitoring of the URLs. All the web servers hosted application, load balancer that is serving the application or Application server that hosted the web application must be checked randomly or every day before business hours starts.
To do this monitoring team started working on listing out all loadbalancer URL, primary, secondary webserver URLs and also Application server hosted URL lists and checking on the browser each URL. Here is the pain-area doing this in manually there could be
Human errors, Re-testing same URL 
Each URL checking takes almost 1 min
Business have 10 domains then each domain have load balancer URL, primary, secondary webserver URL approx 30 min
Thought of the day! Why don’t we automate this task using UNIX shell scripting. My friend Pubba Praveen came up with idea using wget command.  Started brainstorming searching on internet for a scripts that could monitor the URL. Many forums, blogs, most of them are using curl command whereas Solaris box is not having this command.
Alternate to curl command we have powerful wget command on Solaris.
Challenges
1. While using wget it is downloading junk
2. We have URL list some of them are SSO enabled
The first challenge, can be fix with –spider option will avoid the downloading anything from the website.
The second challenge overcome by using --no-check-certificate option
Redirect output of command into a variable
Test the test url from the command line, that is giving output from HTTP header of that URL. From that we need only single line that shows HTTP status. The grep command will do this, but it cannot be applied directly to that command. Because the wget execution is unable to sending the response to STDOUT, rather it will send to STDERR. To fix this, redirect the STDERR to STDOUT and do this in background ( 2> &1).  You can do this by storing that into a file, but it is not optimum solution. To store it into a variable, So that grep command will be enabled to find the pattern ‘HTTP’.
Create a file with desired URL list - urls.txt
Prepared a bash script that will work as web crawler for your business to check the URL and tells you it’s host is OK or not.
Effective automation script made the script as follows:

#!/bin/bash
#=========================================================
# This script will check for the appliction url working or not
# Author: Pavan Devarakonda
# Date  : 5/29/2012
#=========================================================

# Function that check the URL returns HTTP Status Code
checkURL()
{
       # keep below 2 lines in one
       s=`(/usr/sfw/bin/wget -t 0 --spider --no-check-certificate $1) 2>&1 \
       grep HTTP| tail -1|cut -c 41-43`
       echo $s
}

#========== = = = main script = = = ===========
clear
for url in `cat urls.txt`
do
         HTTPCode=$(checkURL $url) # function call
         if [ $HTTPCode -ge 500 ]
         then
             status="DOWN"
             # mail if this have issue
         else
             status="UP"
         fi
         TIME=`date +%d-%m-%Y_%H.%M.%S`
         echo At $TIME the URL $url $HTTPCode $status
done

This script can be associated with schedule job
 1. crontab old way
2. autosys job scheduler
3. at command for trials
 
Now concern is when my webserver is down the script hangs..
Looking for help now... any comments are invited

Wednesday, February 2, 2011

Multi server code Backup

If you digest Operating Systems basics or SHELL basics, you might think much effectively and you can make  more optimized ways to leverage your work, you can design your script with enhanced ability with productivity. Once you taste that flavor in my scripts, you can feel that and understand that why did I told those exaggerated words. Let me share my experience on this.

In every J2EE production environment is going to have an application code (most of the developer say it the composition of 'dist' and 'properties'), which could be no-stage mode deployment style. Where you need to have a copy of code on every WebLogic server running machine. For every production release there could be success or fail. A wise WebLogic Administrator always takes the backup of the existing code and then go further to do fresh code deployment.

According to the best practices of WebLogic deployment process, First time you will do the sample test on the production environment by making single site available for test. That is if your application having EJB-Tier, Web-tier then, start each one server in the tier. Test the basic flow test, which makes you know wheather your new code works or not. Most of the cases the code come to production move is already checked on Staging environment which is the simulated environment of production environment.
It is always good to do double check your work before you do something on Production environments.

Here the important point is that to overcome failure situation, when a new changes in the code fails , such cases alternate solution is to revert back the old code.

How to do this?
Think!! Think wisely, Think for optimizing, Think for better outputs...

We can do simple copy command but what makes error-less work
Take backup with date and time stamp.
There could be multiple application codes on the same machine.
By accomplishing these above task with the following script on a machine.
clear
if [ $# -gt 0 ]; then
 today=`date "+%b%d_%y"`
 APPNAME=$1
 SRC=$HOME/$APPNAME
 BKP=$HOME/backups/$APPNAME
 mkdir -p $BKP
 BKP=$HOME/backups/"$APPNAME"_"$today"
 echo $SRC
 echo $BKP
  
 echo '1. Code backup [dist]only'
 echo '2. Full backup [includes dist, properties]'
 read -p "Please enter your option [1/2]:"  opt
       if [ $opt -eq 1 ] || [ $opt -eq 2 ]; then 
  case "$opt" in
  
 1)  echo "copying dist ..."
  cp -rp $SRC/dist $BKP
     ;;
 2)  echo  "copying all including properties..."
  cp -rp $SRC/dist $BKP
  cp -rp $SRC/properties $BKP
     ;;
 esac
        else
                echo "  invalid option "
        fi
else
    echo "Please enter application name in command line arguments"
fi


Multiple site backup
The above backup is required for every site (multiple machines). If you have already used ssh key generated and all sites are accessible without password. In Ramayan Ram's single arrow hits seven trees in a row. Similarly I want to make the backup script run on one machine and hit on every machine where the application is running. But to make this idea come true I took the help of LinkedIn discussion. Mr. John suggested to keep CMD and execute ssh command.

clear
today=`date "+%b%d_%y"`
if [ $# -gt 0 ]
then
        APPNAME=$1
        SRC=$HOME/$APPNAME
        BKP=$HOME/backups/"$APPNAME"_"$today"
        echo '1. Code backup only'
        echo '2. Full backup [includes properties]'
        read -p "Please enter your option [1/2]:" opt
        if [ $opt -eq 1 ] || [ $opt -eq 2 ]
        then
                CMD="mkdir -p $BKP"
                case "$opt" in
                1 )
                        echo "Will copy only dist."
                        CMD="$CMD ;    cp -rp $SRC/dist $BKP ;     ls -lrtR $BKP ;   exit"
                        ;;
                  2 )
                        echo "Will copy dist and properties."
                        CMD="$CMD ;    cp -rp $SRC/dist $SRC/properties $BKP ;    ls -lrtR $BKP ;   exit"
                        ;;
                 esac
  # serverlist is the text file
                for i in `cat serverlist`
                do
                        echo "Connecting to $i."
                        ssh -a $i $CMD
                done
                echo "done !"
           else
                echo " invalid option "
          fi
else
        echo "Please enter application name in command line arguments"
fi

This script worked... awesome

Blurb about this blog

Blurb about this blog

Essential Middleware Administration takes in-depth look at the fundamental relationship between Middleware and Operating Environment such as Solaris or Linux, HP-UX. Scope of this blog is associated with beginner or an experienced Middleware Team members, Middleware developer, Middleware Architects, you will be able to apply any of these automation scripts which are takeaways, because they are generalized it is like ready to use. Most of the experimented scripts are implemented in production environments.
You have any ideas for Contributing to a Middleware Admin? mail to me wlatechtrainer@gmail.com
QK7QN6U9ZST6