Configuring Plain NodeManger on Solaris

Today morning one of my buddy pinged me from my team, seeking help on an issue he is faceing in his newly  configured WL domain for his learnings.

Understanding the Issue
My buddy novice WLA is trying to start the WebLogic instance on a remote machine. He is working on Solaris 10 operating environment, WebLogic 9.2. I asked for Java version, He executed the following command:

$ java -version

I asked for jps command execution the system is refused and said that ' jps is bad cammand'. I understand that  JAVA_HOME is not set for his environment. asked for echo $PATH. It is showing Java Path. Ohh!! what is missing then??

$which java
this resulted that JAVA_HOME is NOT set to weblogic installation JDK, it was pointed to JAVA_HOME come with the Solaris installation.  I understand that could be one of the reason of not starting the server instance. Update the .profile file with  JAVA_HOME as required to set JDK residing in WebLogic installation path.

What is your NodeManager saying?
My buddy located the logs and found the following line
javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE

To avoiding this Exception you must have certificate generated and installed as requested bye nodemanager. Why we need this SSL certificate? Open the WebLogic Admin console, found that he is working with SSL type of Node Manager. If you don't have secure certificates to configure you should not use SSL type nodemanager. I suggested better to use "Plain" type for the NodeManager configuration on the machine for learning purpose.  Change it in the admin console and save the changes.

Redefining NM properties

You need to update few NodeManager properies on every machine your domain is configured.

Machine wise changes
ListenAddress=yourmachineip
ListenPort=5557

Common "Plain" setting for all the machines involved in domain
SecureListener=false

Stop the Admin server and managed servers (if any running) stop the NodeManger in all machines the domain configured.


Now, First start the NodeManager in the which Admin server residing then start the Admin Server. After Admin Server starting all the domain configuration details will be captured by NodeManager on that machine and ready for communicate with remote NodeManagers. Next, Start the NodeManager on the remote Machine(s) . 

Finally, start the remote server using admin console.  to start your remote managed server make sure that your admin server, Node Managers on admin machine and remote machines must be running without any ERROR or WARN messages. Conclude your configuration is successful by checking the Node Manager logs, and on the admin console check the server state

Note that jps command will list all java processes which includes NodeManager process too.
$jps
12121 NodeManager
4544 Server

Further References for Secure NodeManager:
1. SSL Node Manager configuration
2. Fisal SSL

Best Practices for WebLogic Environment

Here I am jotting out few interesting Best practices for Oracle WebLogic environments, which I have experienced/encountered hurdles while preparing a WebLogic Domain. To Win this Running race you must overcome these hurdles, the best solutions is remembering all of them now I am sharing with you guys here:

1. Dedicated User and group
Oracle WebLogic installation on Solaris machine or Linux or a Windows machine, it is better to have a dedicated user and shared growup where you can install the Middleware components WebLogic, Coherence, WebCenter sites, Content Management etc. provide access to all  so that all other users need not to installing  for each new domain on the same machine.

  useradd [options] LOGIN

Some of important options are:

-d home directory
-s starting program (shell)
-p password
-g (primary group assigned to the users)
-G (Other groups the user belongs to)
-m (Create the user's home directory

My experiment:

useradd -g wladev -s /bin/bash -p xxxxxxxx -d /home/wladmin -m wladmin

Remember that, You can run above user creation command if you have root user access only. Double check the password working for the newly created user. Now a days Virtual Box users are becoming super user (root) just by sudo access. Switch to user (su - wladmin) will connect to the new user.
Change the user password from root user, using passwd wladmin command. On the root user it won't ask you previous password.

2. Using of sed for Migrations
When I worked for WebLogic 8.1 to WebLogic 9.2 migration for each instance wise configuring fresh properties updating took me around a week time for whole environment. Time changed and the requirements changed and this time WebLogic 11g migration from WebLogic 9.2 I have an idea to use stream editing option, and applied with proper regular expressions to finish my task. It worked for me perfectly it is awesome, whole updating done in half hour with small script that included sed in a for loop. I had experianced the fastness  with sed to change multiple lines search and replace in multiple files in the same machine

The following diagram will tells you how sed works on text patterns.

SED Script functionality

Learning SED scripting

for i in `ls instances|grep c`
do
cd $INST_HOME/$i/config
cp /oldinstance/$i/config/*.properties .
sed -e "s/$i-//g" \
      -e "s/$i\_//g" \
      -e "s/\_$i//g" \
      -e "s/-$i//g" \
      -e "s/wluser92/wluser11/g"  <  log4j.properties  >temp
mv temp log4j.properties

User per domain: If you are preparing a development environment then you can choose a user per domain it is the best way to avoid conflicts between developers code changes etc. Install new Oracle License and keep always a backup of old License.

3. Customizing your domain

AdminServer name, ListenAddress, ListenPort, Some times you might see errors saying that "Listen Port already in use", To avoiding port conflicts: Before assigning a port to your WebLogic instance better you check whether it is already in use or not? by using netstat command.
About Virtual IP issues.
i) WebLogic Server wise logs generation
WebLogic Server instance each one can generate separate server side STDOUT logs, STDERR logs as well as application logs. These logs must be collected in a separate mount point will make free for disk utilization memory problems. According to the application severity we can keep archiving the rotated logs on the disk. Most of the Admins, developers while doing troubleshooting for an issue they must revisit these logs and they must know from which server it was happening for this log4j provides more flexibility to digg/debug every Java package, class level, even method, line level too.

How to collect it?
To make this possible you need to enable your WebLogic server library path must pick the log4j-1.2.8.jar and the logging definitions in a separate file lets say it as log4j.properties file in the CLASSPATH.

Where to set?
Before weblogic.jar path or after? Oracle recommands application related jars and third-party jars must be set after weblogic.jar. So log4j.xx.jar must be in POST_CLASSPATH.

ii) Editing for all Domain Environments

a. JAVA_OPTIONS
b. USER_MEM_ARGS
c. JVM type
d. CLASSPATH (PRE/POST)
e. Native IO options
f. MuxerThread
g. SocketReaders

Changing JVM Hotspot Compiler

Editing common scripts impacts all the domains in that machine. If there is a need for the WebLogic server run with server JVM that will give more scope for In commEnv.sh script we are going to update .
Sample Example:

191    Sun)
192      JAVA_VM=-client
change to
191    Sun)
192      JAVA_VM=-server

d
4. Scalability for Domain
Adding servers to  or removing servers existing Cluster is nothing but scalability. While defining new Cluster identify proper multi-cast address suitable to your environment with multi-cast test. You must use Interface Address as the DNS/IP of the machine where the instance is configured in each server's Cluster tab. This will make you easy to run the clustered environment.

5. Effort saving means cost saving
While preparing your configuration keep focus on portable coding. I have been to many UNIX forums to find portable and flexible scripting. Here I am sharing for you. Efforts can be focused on the following things

Customized Stop All script for each domain 
Best option I found from stopManagedWebLogic.sh given by the Oracle. It is normal shell script it will invokes the Python script on the fly. I just replaced the 'Server' with 'Cluster' argument for shutdown WLST command and also called admin stop script in the bottom.

#!/bin/sh

DOMAIN_HOME="/home/domains/mydomain"

. ${DOMAIN_HOME}/bin/setDomainEnv.sh
ADMIN_URL="t3://my.adminhost.com:adminPort"

echo "wlsUserID ='username'" >>"shutdown.py" 
echo "wlsPassword = 'password'" >>"shutdown.py" 
echo "connect(${wlsuserID}, ${wlsPassword}, url='${ADMIN_URL}')" >>"shutdown.py" 
 
#=== fetching cluster list from the domain configuration repository =============
for cl in `grep -i clstr\<\/n ../config/config.xml|sed  's/.*\(.*\)<\/name>.*/\1/'
do
 echo "shutdown($cl','Cluster')" >>"shutdown.py" 
 echo "state($cl)">>"shutdown.py" 
done
echo "shutdown()" >>"shutdown.py" 
echo "exit()" >>"shutdown.py" 
 
echo "Stopping Weblogic Server..."
java -classpath ${FMWCONFIG_CLASSPATH} ${MEM_ARGS} ${JVM_D64} ${JAVA_OPTIONS} weblogic.WLST shutdown.py  2>&1 

echo "Done"

iv. Server Health checking script
Separate is a time, effort saving

6. Deployment Strategies
System resource deployments better you prepare your customized JDBC Data source configuration script using WLST. Avoid Start-up Classes configuration which will make dependable deployment, which leads you to not able to use side-by-side(SBS) deployment advantage.

7. WebLogic 9.x onward you have a flexibility to use Deployment plans for UAT, QA, Staging environments as same as production. This will reduce the problem of porting the code on one environment to other without any configuration changes.

8. OutOfMemoryError

Most of the production environments first hurdle is OOME, if it is occurring in your Web-tier environment then you can use JSP pre-compile.Prepare GC monitoring scripts

9. Now a days everywhere you can find Virtualization (SOLARIS ZONES), Cloud computing, Clustering (Database RAC), Grid concepts, Veritas Clustering in Disks/RAID etc. Have each topic in breif knowledge that could make you understand if anything goes wrong somewhere in the application environment you can easily figure it out.

10. Know about your production environment end to end how the data flows? Firewall, Load balancer software or hardware, proxy-plugin security aspects, Network connectivity, Net-backup locations etc.

Reference URL:
1. Best Virtual IP usage http://blogs.oracle.com/muralins/2007/08/ipmp_ip_multipath.html
http://www.eng.auburn.edu/~doug/howtos/multipathing.html
2. Linux Administration Commands http://www.faqs.org/docs/abs/HTML/system.html
3. WLST Configuration help http://wlstbyexamples.blogspot.com/
4. WebLogic Upgrade http://download-llnw.oracle.com/docs/cd/E13179_01/common/docs100/upgrade/comm_ref.html

Do you have alias in your profile??

Making mistake is human nature but, overcoming that mistake turning to desired outcome is very wise thing. You want to learn about that wise trick?


If yes follow me!!

In Unix platform there is a file called .profile (some Linux enviroments it is .bash_profile). Which executed automatically when you logon to your user, which inturn have the enviroment variables which are required to run your application on WebLogic. For WLA required to setup a standard .profile for his production enviroment that is for JAVA_HOME, WL_HOME are major some of the enviroments requires ORA_HOME in this file.

Defining JAVA_HOME, WL_HOME, etc is common thing for any WLA. But your my Smart WLA then you might applying a intersting UNIX command 'alias'. Most of the time we use 'ls' command for listing the files/folders. By mistake you might mistype as 'sl' instead of 'ls'. sometimes feeling hungry!! or thursty!! and typing command that may go wrong like 'cd..' or 'cd...'. Collect all these common mistypos and make use of 'alias' command then see!!

Though, You make mistake in typing it will give you desired command output. I like this command, since when I found it. This command makes magic, it is very interesting feature in UNIX. Just append to your .profile the following lines:

######WLA Smart idea for using alias #############################
alias cls=clear
alias sl=ls 
alias cd..="cd .."
alias cd...="cd ../.."
alias gi="grep -i"
alias l="ls -al"
alias lm="ls -al | more"
alias h=history
alias hm="history | more"

This is value added to your work, obviously it will reduce your time spending on working machine. Performance will be improved if you find more common mistakes in your enviroment and sortout with 'alias' the magical command, it will give you fruties results!!

WebLogic Port mapping to Process ID

Hey dear WLA most of us come from Development environment to production environments. Hope the scienario thatI am going to discuss here is a common to everyone, who is working on development environment. Sometime or other you might felt I should have a handy script that could takes input as a WebLogic instance port which usually get from the applicaiton URL. The WebLogic port and list all the Process IDs which are associated with it on UNIX(here I got solution for Solaris) environment. Recently I found a Sun blog which is clearly discussed similar issue. I am re-compiling the same with customizing to our WebLogic Listen Port, WebLogic Server runs with a Java generated child Process ID. 'lsof' is the command you can execute and get this solved but most of the times it is a Sun third party downloads, that is not allowed in many Banking, Financial and Insurance Service organizations. And one more reason is small companies doesn't efford for third party tools.

Then, Your choice will be writing a handy script that will do the same task as lsof command helps to find the process id for a given WebLogic Listening port.

#!/bin/ksh 
 
pids=$(/usr/bin/ps -ef -o pid=)
 
if [ $# -eq 0 ]; then 
   read wlport?"Enter port you would like to know Java Process Id for: " 
else 
   wlport=$1 
fi 
 
for f in $pids 
do 
   /usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $wlport$" 
   if [ $? -eq 0 ]; then 
        echo "===============***=============***==============="
        echo "ListenPort: $wlport is being used by Java PID:\c" 
        ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f 
        exit 0 # if you suspect more Weblogic instances with same listen port remove this
   fi 
done

Note: Save this script to commonly accessing user location name it as WLPort2Pid.ksh
Run with argument or without also it will work!! But you need to input the listening port that is must.
-->

Writing about this experiment is a new learning for me too.
Hope you enjoyed this one!! HAPPY SCRIPTING!!