Oracle WebLogic Server: Advanced Administration

Installation Management Smart Update Patch Management Domain Templates Domain Templates Extension Templates Template Builder Other Domain Tools WLST and Templates Pack Tool Configuration Backup Configuration Auditing Advanced Network Configuration Network Channels Administration Channels Standby Mode Multi Data Sources Data Source Failover Data Source Load Balancing Connection Testing WebLogic and RAC JDBC Performance Essentials Connection Pooling Statement Caching Last Logging Resource JMS Message Management Viewing Messages Exporting Messages Pausing Destinations JMS Guaranteed Messaging Persistent Stores Durable Subscribers

JMS Performance Essentials Quotas Thresholds Unit of Order JMS Store and Forward SAF Agents SAF Contexts SAF Imported Destinations JMS Message Bridge Bridge Adapters Bridge Destinations Quality of Service Server Migration Cluster Leasing Candidate Machines JMS Clustering JMS Cluster Targeting Service Migration Migratable Targets Distributed Destinations JMS Load Balancing Cross-Cluster Replication MAN Replication WAN Replication Authentication Providers Security Realms Security Stores Control Flags LDAP Authenticators SQL Authenticator Password Validation Security Migration Server Performance Essentials Load Testing JVM Tuning Heap Monitoring Work Managers Monitoring and Diagnostics Essentials Runtime MBeans Diagnostics Framework Metric Collectors Watches Instrumentation WebLogic and SNMP Trap Monitors

WebLogic Server: Performance Tuning

Introduction to Performance Monitoring Overview of Performance Methodology Importance of Performance Tests and Benchmarks Introduction to Load and Stress Test Tools Configuring the Grinder Introduction to Oracle Application Testing Suite Monitoring and Tuning JRockit JVM Overview of Java Virtual Machines Understanding Garbage Collection and Heap Fragments JRockit JVM Benefits JRockit Mission Control JRockit Runtime Analyzer JRockit Memory Leak Detector Tuning JRockit JVM Garbage Collection Configuring JVM Parameters Tuning Hotspot JVM Overview of Hotspot JVM Tools for Monitoring Hotspot JVM Using Java Visual VM Command-line Tools to Monitor Hotspot JVM Understanding JVM Ergonomics Use of Throughput Goal Use of Footprint Goal Configuring JVM Parameters Configuring Work Managers Introduction to WLS Self-tuning What is Work Manager? Threadpool and Priority Scheduling Guidelines Work Manager Configuration Work Manager Scope and Sharing Constraints Introduction to CommonJ Configuring Other WebLogic Server Resources Domain Startup Mode Native IO Performance Stuck Threads Tuning Chunk Size Connection Backlogs

Using the JSP Compiler jspc Using the Precompile Option Setting JSP Page Check Interval Setting Servlet Reload Check Interval Defining WebLogic Cache Tag Tuning JDBC Tuning Connection Pools Tuning Statement Caches Performing Batch Updates Tuning Transactions Tuning Database Specific Parameters Tuning EJB Tuning Pool Size Tuning Cache Size Tuning Concurrency Strategy Tuning Entity Bean Parameters Tuning Transactions Tuning EJB Clients Tuning JMS Tuning JMS Clients Defining Aggregation and Message Pipeline Describing Persistence Techniques Listing Throttling Techniques Considerations for Store and Forward message Tuning WebLogic Server Clusters Architectural Consideration Load Balancers and Firewalls Session Persistence General Tuning Tips

Oracle WebLogic Server: Troubleshooting Methodologies

Introduction to Diagnostic Patterns Overview of Patterns What is a Pattern Identifying Support Pattern Methodology Why Use Support Patterns Identifying Support Pattern Resources Common Diagnostic Tools JVM Management: Java SE 6.0 Monitoring and Management Architecture Identifying Processes and Threads Obtaining a Thread Dump Using WLS Memory: Define Java Heap Garbage Collection Review Configuring JRockit Heap Server Core Dump Pattern Identifying Server Core Dump Pattern Symptoms and Causes of a Server Core Dump Identifying Why Core File is Not Produced Investigating Causes of Server Core Dump Overview of Investigation Overview of Core File Tools Overview of Standard Investigation Tools Creating Troubleshooting Checklist Generic Server Hang Pattern Symptoms and Causes of a Server Hang Investigation of Server Hang Basic Investigation Steps View the Execute Threads Thread Dump on a Running Server Analysis and Troubleshooting Investigation Results Setting Server Thread Counts Troubleshooting Strategy Identifying High CPU Usage Pattern and Describing Resolution Steps Identifying High CPU Usage Pattern Review: Processes and Threads Investigating High CPU Usage: On Solaris, Linux, and Windows OS Analyzing the Server Instance to Determine What Threads are Causing the Issue Using the Analyzed Information to Identify Areas Where Performance Tuning is Required Analyzing and Diagnosing Out of Memory/Memory Leak Patterns Review of General Memory Concepts: Java Heap, Native Memory, and Memory Leaks Investigating Java Heap Out of Memory Errors Review Garbage Collection and Object References Symptoms, Causes, and Troubleshooting of Out of Memory in Java Heap Investigating Out of Native Memory Conditions Review: Process Size, Virtual vs. Physical Memory Symptoms, Causes, and Troubleshooting of Native Out of Memory Conditions

Generic JDBC Pattern Understanding Generic JDBC Pattern Review: JDBC Database, Datasources, Connection Pool, and Statement Cache Investigating JDBC Problems Server Startup: Failed Pool Creation Causes of Resource Exceptions Overview of Insufficient Connections Troubleshooting WebLogic Server Crashes Troubleshooting WebLogic Server or Application Hangs and Memory Leaks Troubleshooting Too Many Open Files Pattern Symptoms, Causes, and Troubleshooting of Too Many Open Files Problem Identifying a WebLogic System That Has Too Many Files Open Using Available Tools to Determine Which Component Has Too Many Files Open and What the Files Are Describing and Diagnosing Class Exception Patterns Describing Class Cast Exception Pattern Investigating Class Exceptions and Errors Adding Application Code Diagnostics Identifying Different Class Loaders Describing Known WLS Issues Understanding and Troubleshooting Class Casting Issues Understanding and Troubleshooting Basic Class Loader Issues Understanding Symptoms, Causes, and Tools to Troubleshoot Multicast Patterns Reviewing a WebLogic Server Cluster and Clustered Server Communication Describing Multicast Problems and Investigation Configuration Errors and Physical Network Problems Multicast Timeouts Clustering Heartbeat Problems and Multicast Storms Testing and Debugging Multicast Patterns Discussing Some of the Potential Multicast Issues that Can Occur in a WebLogic Server Cluster Understanding the Processes Involved in Identifying and Resolving the Issues General Diagnostics for Proxy Plug-ins Pattern Diagnosing and Troubleshooting Proxy Plug-ins Pattern Gathering Plug-in Diagnostics Switching on Debugging for the WebLogic Server Proxy Knowing Where to Look in the WebLogic Server Console and Proxy Logs for Issues Identifying, Investigating, and Diagnosing HTTP Session Replication Failures Pattern Reviewing HTTP Session Replication Failures Pattern Causes, Symptoms, and Examples of HTTP Session Replication Failure Investigating HTTP Session Replication Failures Gathering Diagnostics: Enable Debug Understanding How the WebLogic Server Proxy Server Works with Respect to HTTP Session Replication and Failover Discussing Various Aspects of HTTP Session Replication Configuration and Operation

String Concatenation in Python

Build a list of strings, then join it

def method():
 str_list = []
 for num in xrange(loop_count):
   str_list.append(`num`)
 return ''.join(str_list)

This approach is commonly suggested as a very pythonic way to do string concatenation. First a list is built containing each of the component strings, then in a single join operation a string is constructed conatining all of the list elements appended together.

Write to a pseudo file

from cStringIO import StringIO
import timing, commands, os
from sys import argv
def method():
 from cStringIO import StringIO
 file_str = StringIO()
 for num in xrange(loop_count):
   file_str.write(`num`)

The cStringIO module provides a class called StringIO that works like a file,
  but is stored as a string. Obviously it's easy to append to a file - you simply
  write at the end of it and the same is true for this module. There is a similar
  module called just StringIO, but that's implemented in python whereas cStringIO
  is in C. It should be pretty speedy. Using this object we can build our string
  one write at a time and then collect the result using the getvalue() call.

 return file_str.getvalue()

downloading rpm packages without installing them

You need to install plugin called yum-downloadonly. This plugin adds a --downloadonly flag to yum so that yum will only download the packages and not install/update them. Following options supported by this plugin:

[a] --downloadonly : don't update, just download a rpm file
[b] --downloaddir=/path/to/dir : specifies an alternate directory to store packages such as /tmp

How do I install yum-downloadonly plugin?

Type the following command to install plugin, enter:
# yum install yum-downloadonly



After this plugin is installed on yum, you can use the following command to download any rpm package and store it to downloaddir location.


Download RPM Packages
yum reinstall libXtst --downloadonly --downloaddir=/tmp

Identify the Java process hogging all the CPU

Identify the Java process hogging all the CPU

To do that, use the top or prstat command to get that process ID

$ top
last pid: 25837; load averages: 0.06, 0.15, 0.36 16:14:18
73 processes: 72 sleeping, 1 on cpu
CPU states: 0.0% idle, 99.1% user, 0.9% kernel, 0.0% iowait, 0.0% swap
Memory: 4096M real, 1191M free, 2154M swap in use, 1306M swap free

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
1864 rdadmin 53 59 0 184M 88M sleep 7:38 98.10% java
27794 iwui 39 59 0 222M 186M sleep 882:38 0.42% java
27708 iwui 71 59 0 52M 34M sleep 82:40 0.06% java
24025 root 29 29 10 232M 146M sleep 716:03 0.04% java
23449 rdadmin 1 59 0 2616K 2112K sleep 0:00 0.02% bash
287 root 13 59 0 6736K 5560K sleep 110:25 0.01% picld

$ prstat
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1864 rdadmin 184M 88M sleep 59 0 0:07:39 98.10% java/53
27794 iwui 222M 186M sleep 59 0 14:42:46 0.9% java/39
26030 rdadmin 4592K 4200K cpu2 49 0 0:00:00 0.1% prstat/1
27708 iwui 52M 34M sleep 59 0 1:22:41 0.1% java/71
24025 root 232M 146M sleep 29 10 11:57:18 0.1% java/29
287 root 6736K 5560K sleep 59 0 1:50:25 0.0% picld/13
15686 oemadmin 107M 93M sleep 29 10 1:13:19 0.0% emagent/5
15675 oemadmin 8096K 7432K sleep 29 10 0:12:34 0.0% perl/1

Depending on the number of CPU in the server and the number of CPU hogging thread. So if there is only 1 thread hogging cpu on a 2 processors server, the process associate with that thread will consume 50%.

2- Identify the thread ID that are hogging all CPU

To do that, we will use the prstat command with a special switch:

bash-2.03# prstat -L
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1864 rdadmin 985M 577M cpu2 0 2 0:58.24 22% java/84
1864 rdadmin 985M 577M cpu0 0 2 0:51.38 22% java/138
1864 rdadmin 985M 577M cpu3 0 2 0:50.23 22% java/122
1864 rdadmin 985M 577M run 0 2 0:58.41 21% java/145
26430 tomcat 877M 456M sleep 52 2 0:09.27 1.2% java/9
26399 tomcat 985M 577M sleep 52 2 0:11.09 0.9% java/9
26461 tomcat 481M 324M sleep 52 2 0:06.36 0.7% java/9
26430 tomcat 877M 456M sleep 47 2 0:00.11 0.5% java/174
26430 tomcat 877M 456M sleep 28 2 0:00.09 0.3% java/93
26492 tomcat 460M 300M sleep 52 2 0:06.01 0.2% java/9
26430 tomcat 877M 456M sleep 22 2 0:00.14 0.2% java/129
26461 tomcat 481M 324M sleep 32 2 0:00.04 0.2% java/104

Here you can see 4 threads that are using all of the machine cpu.

3- Get the real thread ID

For an unknown to me reason, prstat shows only the LWPID (lightweigth process), see at the end of a prstat output line. We now need to find the real thread ID. Generally, they are the same, but they may be different too. To do that, use the pstack command. Be sure to be logged with the process owner or root. Grep the pstack output to only get the significant lines:

$ pstack 1864 | grep "lwp#" | grep "84"
----------------- lwp# 84 / thread# 83 --------------------

The real thread ID is 83

Now convert that number in hex: 53

4- Get the Java process stack real time

Now comes the interesting part, get the Java process stack for all thread. To do that, you need to send a special signal to the process using the kill command (don't worry, this will not kill the process, only force the JVM to dump its thread stack to the default output log).

We will do this 2 commands at a time, to be sure to get the output from the log. So first change your current directory to tomcat log directory (for a tomcat process).

$ cd /applications/tomcat/servers/tomcat1_rd1/logs

Then send the a signal #3 to the Java process to force it to dump its thread stack, then tail the log.

$ kill -3 1864 ; tail -f catalina.out

Wait maybe 1-3 seconds then you will se a large output be added to the log:

Look at the stack and locate the nid=0x### which is your thread ID in hex (53 in the example)


TP-Processor24" daemon prio=10 tid=0x00a0b180 nid=0x53 runnable [0xb3e80000..0xb3e81a28]
at java.util.ArrayList.indexOf(ArrayList.java:221)
at java.util.ArrayList.contains(ArrayList.java:202)
at com.company.content.business.ContentHandler.getRandomContent(ContentHandler.java:248)
at com.company.rd.action.CycleContentTypeAction.execute(CycleContentTypeAction.java:100)
at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:480)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:274)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1420)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:502)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.readersdigest.content.filter.MemberContentBundleStatusFilter.doFilter(MemberContentBundleStatusFilter.java:319)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.readersdigest.rd.filter.GenricRequestValuesFilter.doFilter(GenricRequestValuesFilter.java:94)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.readersdigest.rd.filter.BreadcrumbFilter.doFilter(BreadcrumbFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.readersdigest.rd.filter.RdAutoLoginFilter.doFilter(RdAutoLoginFilter.java:71)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.readersdigest.servlet.filters.GrabTrackingParametersFilter.doFilter(GrabTrackingParametersFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.readersdigest.servlet.filters.HibernateFilter.doFilter(HibernateFilter.java:78)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.readersdigest.servlet.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:134)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:307)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:385)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:748)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:678)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:871)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)

You now have a running thread stack to investigate possible infinite loops or something else that is hogging all CPU.