Best Practices for WebLogic Environment

Here I am jotting out few interesting Best practices for Oracle WebLogic environments, which I have experienced/encountered hurdles while preparing a WebLogic Domain. To Win this Running race you must overcome these hurdles, the best solutions is remembering all of them now I am sharing with you guys here:

1. Dedicated User and group
Oracle WebLogic installation on Solaris machine or Linux or a Windows machine, it is better to have a dedicated user and shared growup where you can install the Middleware components WebLogic, Coherence, WebCenter sites, Content Management etc. provide access to all  so that all other users need not to installing  for each new domain on the same machine.

  useradd [options] LOGIN

Some of important options are:

-d home directory
-s starting program (shell)
-p password
-g (primary group assigned to the users)
-G (Other groups the user belongs to)
-m (Create the user's home directory

My experiment:

useradd -g wladev -s /bin/bash -p xxxxxxxx -d /home/wladmin -m wladmin

Remember that, You can run above user creation command if you have root user access only. Double check the password working for the newly created user. Now a days Virtual Box users are becoming super user (root) just by sudo access. Switch to user (su - wladmin) will connect to the new user.
Change the user password from root user, using passwd wladmin command. On the root user it won't ask you previous password.

2. Using of sed for Migrations
When I worked for WebLogic 8.1 to WebLogic 9.2 migration for each instance wise configuring fresh properties updating took me around a week time for whole environment. Time changed and the requirements changed and this time WebLogic 11g migration from WebLogic 9.2 I have an idea to use stream editing option, and applied with proper regular expressions to finish my task. It worked for me perfectly it is awesome, whole updating done in half hour with small script that included sed in a for loop. I had experianced the fastness  with sed to change multiple lines search and replace in multiple files in the same machine

The following diagram will tells you how sed works on text patterns.

SED Script functionality

Learning SED scripting

for i in `ls instances|grep c`
do
cd $INST_HOME/$i/config
cp /oldinstance/$i/config/*.properties .
sed -e "s/$i-//g" \
      -e "s/$i\_//g" \
      -e "s/\_$i//g" \
      -e "s/-$i//g" \
      -e "s/wluser92/wluser11/g"  <  log4j.properties  >temp
mv temp log4j.properties

User per domain: If you are preparing a development environment then you can choose a user per domain it is the best way to avoid conflicts between developers code changes etc. Install new Oracle License and keep always a backup of old License.

3. Customizing your domain

AdminServer name, ListenAddress, ListenPort, Some times you might see errors saying that "Listen Port already in use", To avoiding port conflicts: Before assigning a port to your WebLogic instance better you check whether it is already in use or not? by using netstat command.
About Virtual IP issues.
i) WebLogic Server wise logs generation
WebLogic Server instance each one can generate separate server side STDOUT logs, STDERR logs as well as application logs. These logs must be collected in a separate mount point will make free for disk utilization memory problems. According to the application severity we can keep archiving the rotated logs on the disk. Most of the Admins, developers while doing troubleshooting for an issue they must revisit these logs and they must know from which server it was happening for this log4j provides more flexibility to digg/debug every Java package, class level, even method, line level too.

How to collect it?
To make this possible you need to enable your WebLogic server library path must pick the log4j-1.2.8.jar and the logging definitions in a separate file lets say it as log4j.properties file in the CLASSPATH.

Where to set?
Before weblogic.jar path or after? Oracle recommands application related jars and third-party jars must be set after weblogic.jar. So log4j.xx.jar must be in POST_CLASSPATH.

ii) Editing for all Domain Environments

a. JAVA_OPTIONS
b. USER_MEM_ARGS
c. JVM type
d. CLASSPATH (PRE/POST)
e. Native IO options
f. MuxerThread
g. SocketReaders

Changing JVM Hotspot Compiler

Editing common scripts impacts all the domains in that machine. If there is a need for the WebLogic server run with server JVM that will give more scope for In commEnv.sh script we are going to update .
Sample Example:

191    Sun)
192      JAVA_VM=-client
change to
191    Sun)
192      JAVA_VM=-server

d
4. Scalability for Domain
Adding servers to  or removing servers existing Cluster is nothing but scalability. While defining new Cluster identify proper multi-cast address suitable to your environment with multi-cast test. You must use Interface Address as the DNS/IP of the machine where the instance is configured in each server's Cluster tab. This will make you easy to run the clustered environment.

5. Effort saving means cost saving
While preparing your configuration keep focus on portable coding. I have been to many UNIX forums to find portable and flexible scripting. Here I am sharing for you. Efforts can be focused on the following things

Customized Stop All script for each domain 
Best option I found from stopManagedWebLogic.sh given by the Oracle. It is normal shell script it will invokes the Python script on the fly. I just replaced the 'Server' with 'Cluster' argument for shutdown WLST command and also called admin stop script in the bottom.

#!/bin/sh

DOMAIN_HOME="/home/domains/mydomain"

. ${DOMAIN_HOME}/bin/setDomainEnv.sh
ADMIN_URL="t3://my.adminhost.com:adminPort"

echo "wlsUserID ='username'" >>"shutdown.py" 
echo "wlsPassword = 'password'" >>"shutdown.py" 
echo "connect(${wlsuserID}, ${wlsPassword}, url='${ADMIN_URL}')" >>"shutdown.py" 
 
#=== fetching cluster list from the domain configuration repository =============
for cl in `grep -i clstr\<\/n ../config/config.xml|sed  's/.*\(.*\)<\/name>.*/\1/'
do
 echo "shutdown($cl','Cluster')" >>"shutdown.py" 
 echo "state($cl)">>"shutdown.py" 
done
echo "shutdown()" >>"shutdown.py" 
echo "exit()" >>"shutdown.py" 
 
echo "Stopping Weblogic Server..."
java -classpath ${FMWCONFIG_CLASSPATH} ${MEM_ARGS} ${JVM_D64} ${JAVA_OPTIONS} weblogic.WLST shutdown.py  2>&1 

echo "Done"

iv. Server Health checking script
Separate is a time, effort saving

6. Deployment Strategies
System resource deployments better you prepare your customized JDBC Data source configuration script using WLST. Avoid Start-up Classes configuration which will make dependable deployment, which leads you to not able to use side-by-side(SBS) deployment advantage.

7. WebLogic 9.x onward you have a flexibility to use Deployment plans for UAT, QA, Staging environments as same as production. This will reduce the problem of porting the code on one environment to other without any configuration changes.

8. OutOfMemoryError

Most of the production environments first hurdle is OOME, if it is occurring in your Web-tier environment then you can use JSP pre-compile.Prepare GC monitoring scripts

9. Now a days everywhere you can find Virtualization (SOLARIS ZONES), Cloud computing, Clustering (Database RAC), Grid concepts, Veritas Clustering in Disks/RAID etc. Have each topic in breif knowledge that could make you understand if anything goes wrong somewhere in the application environment you can easily figure it out.

10. Know about your production environment end to end how the data flows? Firewall, Load balancer software or hardware, proxy-plugin security aspects, Network connectivity, Net-backup locations etc.

Reference URL:
1. Best Virtual IP usage http://blogs.oracle.com/muralins/2007/08/ipmp_ip_multipath.html
http://www.eng.auburn.edu/~doug/howtos/multipathing.html
2. Linux Administration Commands http://www.faqs.org/docs/abs/HTML/system.html
3. WLST Configuration help http://wlstbyexamples.blogspot.com/
4. WebLogic Upgrade http://download-llnw.oracle.com/docs/cd/E13179_01/common/docs100/upgrade/comm_ref.html