30 man-moths = 30 men X 1 month

Being in the software development industry I worked in multiple projects and on the way met people with diverse mindsets. I interacted with people from the same industry but from different sectors like marketing, operations, R&D, development and people from the other industries mainly as clients having unique point views on a software development project. In the early years of my career I used to go to the client places to study the requirement accompanied by the senior members. The initial discussions were around topics like the scope of work, features of the system to be built, preferable technologies, an approximate estimation of the work and required span of time to complete the project.

Quite Embarrassing

I remember a specific situation where a similar conversation was going on with couple of representatives from the client company. The topic was on estimation and we proposed a rough estimation for the work as 30 man months (well I cant’ recollect the actual figure right now). One of the associates from the client’s side said, “When you say 30 man months does that mean 30 men can finish the project within 1 month and vice versa?” Well you can not consider all minds in audience to be of the same level.
From his point of view it’s quite a rational question, to us it was utterly unexpected. In an adverse situation like this one of our senior members attempted to explain him a bit about the complexity and dependencies involved in the trail of execution of a project, which helped us in getting out of the discomfiture. I roughly remember the project initially estimated around six months which got extended to two years; an illustration of charismatic management.

Why Can’t We?

Couple of months back there was a topic in the news that some people built a playground in a day in the city of Sanford, Florida. If the chap who asked the question to us years ago, had read this news before participating in the conversation, I am sure he would’ve an even more stalwart argument. Building a playground includes months of preparation, heavy equipment, and most important in the equation, manpower, but somehow they made it possible.
Can we do it to software projects? So that someday I could go back and tell the chap that yes we can turn the equation as
30 man-months = 30 men X 1 month or 1 man X 30 months or even roughly 600 men X 1 day (assuming 20 working days a month).

The Debate

The debate becomes absolutely baseless if we consider this figure as only the manpower effort. Should we revisit our estimation standards, methods and techniques once more? Rather provide an estimate including other factor those are involved. Apart from manpower it engages lots of other resources and dependencies on multiple factors at various phases of a project like

• Stakeholders’ interest on the scope, cost, motivation and foreseeing the future of the product.
• Availability of required infrastructure to execute a project
• Availability of appropriate skills and experiences
• Rate of attrition of the human resources
• Proper strategy to mitigate the risks caused by controlled or uncontrolled conditions
• Effective project management plan and monitoring
• Effective communication to dig out the clarity of the requirement
• Proper analysis of the requirement to map elements from the problem to the solution domain
• Time spent on researches to find technical solutions to unpredicted problems during execution.
• Suitable design of the proposed system
• Time spent on testing and rework.
• Time spent on documentation and user training.
• Constraints in deployment.
  

Pros and Cons

Lack of any of the above factors can lead to a delay in the route to delivery. So far we are far away from the miraculous day when we can have all the factors ready as desired at our fingertips while executing a project. Certain factors stand to be uncontrolled, for example, the clients’ interest on financing, rate of attrition and sometimes the unexpected technical hazards on the infrastructure, specially when you are working across different geographical locations. Rests of the factors are more or less controllable.

Stakeholders

From my past experiences I noticed most projects introduce delay while deciding on the scope and cost, from the stakeholders’ perspective. Once we are in a hurry things can be available right at your footsteps but then we end up with rushing into the next phases missing out essential information and the big picture, which might turn up as a big obstacle in the testing phase when the requirement shows up its new faces. A good place to loose out on time.

Experience and Skill

The human resource and skill set problem does not interfere so much where everyone is able to participate in any of the phases of a project but that happens in very few cases. Work distribution is done on the basis of experience levels and skill types. You can have more aggressive project schedule based on how efficient people you have and that’s a universal truth since the early ages of software development. With the advent of new technologies, frameworks and of course a wide variety of resources available on the internet the requirement of the skill sets are not so bad as was before but in that case we need time to learn new things. Rate of attrition is one of the natural calamities hindering the execution of a project particularly when you are lacking proper backup plan. The risk mitigation plan should take care of that. Documentation, well that’s again a time taking event, might help on the knowledge transfer for a new member.

Project management and Monitoring

Project management plans should address these dependencies and make out a critical path to carry out the project. To allocate resources the plan needs to take care of resources and their skills. This is a critical activity since all the resources do not possess identical skills and experiences. Wish they could have, so it would be far easier to allocate resources across different activities of a project and rotate them anytime as needed. Agile methodologies will surely help in close monitoring and recovering from misdirection fast. The advantage of a member acting in versatile roles surely keeps agile processes one step ahead. A vital strength for the small companies with small set of resources. Sometimes working on a small part of the product for a member might bring a miasma on the entirely of the product and its requirement. It restricts the future oriented thinking for a team of developers. Members of a team following agile sometimes end up working in silos.
Are we looking forward to have mix and match of our old methodologies (e.g. Waterfall, Spiral, Iterative, RUP etc.) and the agile ones (e.g. XP, Scrum, Crystal etc.)?

Study and Analysis

A proper communication during requirement study and analysis phases could save considerable amount of rework during the testing and implementation phase. A little bit of experience, psychology and technical expertise help the players in this phase to get into the heart of the picture that the client/user carries in their mind. After all it’s a human to human communication; thank God we don’t have to deal with psychology with a computer. An expressive analysis document saves a lot of time in design.

Construction

The construction phase usually constitutes of architecture planning, design, development and testing. It involves research on new technologies, designing and construction of components and testing the same. With the coming storm of ideas, products, open-sources and buzzwords it’s hard to select an appropriate technology these days. A project should consider the time for this as well and try to minimize as much as possible. A proper technical guidance of an architect toward a structured approach of development can speed up the development process. Selection of an appropriate technology, a bird-eye view of the design, recognition of patterns and a proper introduction of the same, reusability of pieces, adherence to standards during development can be well controlled by an architect, so that you don’t get face to face with re-factoring or re-designing quite often. Another most important thing is to figure out the expectations on non-functional requirement of the system and make out a plan to address those. People without prior experiences with automation are often reluctant about discussing on the non-functional requirement at the initial phase of study.
Designers are of course confined within the scope sketched out by the analysis but if they try they can still foresee the feasible enhancements of a product and design it to be flexible enough to accommodate the changes in the future. You never know, some of the enhancements can be at your doorsteps during the testing and deployment phases. They can make it expressive enough to ease the implementation for the developers, which minimizes time.

Testing and rework is the most important phase where the product actually gets weighed against the requirement and that’s where the time gets more and more elongated. This is quite a normal phenomenon in all most all projects and the target is to eliminate the rework cause by a misinterpretation of the initial requirement. Preparation of test plans parallel to the requirement analysis often helps.
Sometimes a gap is noticed in development and deployment environments (where your product is installed) causing a delay due to rework or configuration in the deployment phase. The preparation of a concrete deployment plan during the architecture or design phase helps in trimming down the time. Specially in a geographically diverse location, on heterogeneous platforms or a scenario where multiple preexisting systems are involved.

Tangible and Intangible Hurdles

We understand that there are ample of reasons why the time taken for a project can not be shrunken as desired. We have controls over the tangible elements involved in the process of execution, for example,

• setting up infrastructure
• usage of tools in various phases of design, development, testing, deployment

But intangible factors are not entirely controllable such as

• Stakeholders’ interest
• Skill of human resources
• Rate of attrition
• Time to map requirement into features of the proposed system
• Time devoted to research for finding out a suitable software/technology a for a given problem
• Time to think and come out with a proper design applying experiences
• Time taken in grasping a new technology or open-source product to use it during development.

So much of thinking processes involved in the entire path and how do you estimate your thinking process, well we don’t have a “Thought point analysis” (like function point analysis) yetJ. Do we have to evolve with an agile method of thinkingJ? We can even minimize the transformation of the thoughts from the user’s mind to the analyst’s mind and then to the architect’s and the designer’s and the developer’s and so on, but how?

Honey I Shrunk The Kids

The Agile method of thinking is not yet in the picture though but specially being in the software industry we are quite familiar with abstraction. From machine languages (first generation) to assembly languages (second generation), from assembly languages to high-level programming languages (third generation e.g. C/C++, Java, PL/I etc.) and then to English like natural languages like SQL (fourth generation) and finally to the graphical interfaces (fifth generation) like integrated development interfaces (IDE) for planning, design, development and most of other areas like requirement capturing, testing. These tools intake the thoughts from the human minds and translate them to other languages like C/C++, SQL, Java etc. The pains of programming in machine or assembly languages have be removed by third generation languages. IDEs made the pain of writing C/C++/Java codes quite easy. We are even thinking about Domain Specific Languages (DSL) to keep up the movement one step ahead. I look at SQL being a DSL for database operations. The history of software development shows us an approach toward reducing time for faster and easier development process, at the same time standardizing solutions.

So now we have our helper friends as Abstraction and Code Generation. Though the term “code generation” has picked up during the last couple of years with the conceptualization of Model Driven Architecture (MDA), the basic concept was there from the beginning when we learned to design compilers. Only thing is that, we are thinking in a step by step manner to reduce the gap between the user’s requirement and the end product shrinking the time of development. Open sources are a playing the lead role in reduction of cost and speeding up on time with reusable standard solutions.

We are even thinking of automation through out the entire process transformation of thoughts at different phases of development and reaching out for business process management (BPM) and modeling. BPM enables us to grasp the thoughts right from the minds of the user and turn into code. Well not exactly, we still have rooms for an architect, a designer, a developer and a tester. I don’t understand why we are trying to address the domain and the business processes separately from different angles of BPM and DSL. If you are given with a graphic interface to build your process model and generate code out of it, would you take the pain of writing the same using a DSL? A good use of DSL could be defining the tits and bits of the business process after you have modeled and generated it using a BPM tool. A BPM tool is of course a good tool to address the issue integrating your new systems with old ones irrespective of the platform they are built on. Reusing the existing systems is now far easy with the advent of Service Oriented Architecture (SOA) and that too auto-generating the code for that.
We can certainly imagine the day when one can generate the entire solution in a single day sitting on his/her desk alone. I will have to repent on that day for not keeping the contact number of the chap who asked the question year ago.

Multihomed J2EE Clustering: A Case Study

These days Enterprises are going ahead with their plans for clustering as a mandatory factor in their mission-critical applications. There are many articles scattered over the internet on J2EE clustering using several application servers. There is a nice article on J2EE Clustering that gives an overview on the fundamentals of clustering and its implementation. But being unable to locate such an article that describes a step by step instruction and pitfalls for a particular condition like mine, I studied many of them and sorted out a specific roadmap as demonstrated in this article. It might help someone facing a similar situation.

As per my understanding, a cluster is a logical group of application servers that hosts applications transparently like a single entity and provides a continuous availability to the application. It also provides a high scalability of the application based on the increasing user access.

Clustering in McDonald’s

Envision a situation where there are four counters in a McDonald shop. Three customers are getting dished up at three counters. Once another customer enters the fourth counter is ready for serving. This could be a simple example of scalability. We observe a similar scenario in clusters. When all member servers in a cluster are loaded with multiple requests and a new request comes in, the cluster tries to find out a member server that is less busy and routes the request to that. It’s known as load balancing. There are multiple methods used in load balancing e.g. Round Robin, Server Affinity and Weight based etc. Load balancing is accomplished differently for different application resources. For example, the load balancing mechanisms for Servlets/JSPs could be different from EJB.
Supposing a sales-person operating one of the counters gets occupied by some urgency while serving a customer, he/she can hand-over the customer’s request to another sales-person. This could be an instance of Session Replication and failover. In clusters we see that if a severe problem drives one of the member servers to crumple, another member server immediately takes over the requests that were being served by the previous one and starts acting on them in order to keep the client transparent from the failure. By this feature the applications can be made more highly available to the users. Persistence of a session is necessary to replicate the session. Application server vendors use multiple mechanisms to implement Session Persistence e.g. Memory based, Cookie based, File based, JDBC based and In-memory based etc.
Let us go little more inside the shop with the permission of the shop owner (just kidding). The requests were jotted down on a piece of paper (a good example of session persistence) and routed to the chef who cooks a yummy dish for you. Again there could be many chefs operating in that shop (another cluster). Based on who is less busy the request is being routed to him/her (load balancing again). We can clearly see that there are members working in two different layers (sales-person layer and chef layer) in that shop. Similarly we can have two layers in clustering as well – a layer web server cluster and a layer of application server cluster.

Experiment on MyWeb application

In this article I will describe a case study on clustering and try to sketch a blueprint based on my personal experiences.
I got assigned to a project that was requiring an R&D on J2EE clustering to provide scalability and high availability to a commercial web application. The size of the application is nearly about 25mb including static and dynamic contents (mainly Servlets, JSPs and Java beans) and was installed on a single instance of WebLogic server. The application was also using JDBC connection pools and data sources configured for Sybase database. A hardware load balancer was used to route requests to the WebLogic server. There was no failover implemented around the application.

In order to maintain the business secrecy imaginary names and values for the application, servers, machine IP addresses and other resources have been used in this article.
Going further “machines” are referred as the hardware boxes and “logic machines” as a concept used in WebLogic environment.

Goals and assumptions

The main goals behind the experiment was to

1. Implement clustering for the application – “MyWeb” using a web server layer (Apache Httpd) and an application server layer (WebLogic).
2. Deploy using the staged deployment scenario. In this scenario the member servers have their own individual copies of the application being deployed.
3. Replace the hardware load balancer to use web server proxy load balancer.
4. Use multiple machines running on different Operating Systems.
5. Implement the feature of Http session replication as a failover strategy using In-memory session persistence.
6. Separating out the static and dynamic contents at the web and application server layers. This enhances the performance of an application if you have considerable amount of static contents.

Clustering Roadmap

To be able to meet the goals I figured out a roadmap to move ahead. The steps involved are

# Getting the clustered version of WebLogic.
# Figuring out the clustering topology. A plan or blueprint of how the machines, servers, operating systems to be laid out and how the application will be deployed on them.
# Selecting the load balancing mechanism.
# Selecting the participating machines (hardware) for installation.
# Installation of same version of WebLogic on all boxes.
# Configuring WebLogic administration server and WebLogic domain.
# Configuring WebLogic logical machines (representing the physical machines in WebLogic environment or domain) and Node Managers for them.
# Configuring managed instances and assigning them to logical machines.
# Configuring WebLogic clusters and assigning the managed instances to the clusters.
# Installing Apache Httpd as a web server and installing WebLogic proxy plug-in on it and configuring the httpd.conf file to make it point to the WebLogic cluster.

# Starting WebLogic Administration Server, starting node manager on every participating machine and registering them to the administration server.
# Starting the managed instances from the Admin Console.
# Configuring the application servers (admin and managed) with application (to be deployed) specific settings such as creating JDBC connection pools, setting the security policies to a customized one etc.
# Deploying an application e.g. MyWeb on all the member servers together.
# Configuring the servers with application specific settings, if any and starting the application.

Ingredients & recipe

To start with, here is a list of tools, software, machines, operating systems that was involved during this experiment. I arranged for a clustered version of WebLogic 9.2.1 both for Windows and Linux. In fact I banged on a road block for using WebLogic 9.2.1 for Linux and 9.2.0 for Windows. Unfortunately they are rigid enough to refuse the communication among the member servers with different versions. Apache Httpd 2.0.59 (no_ssl version) can be downloaded from the Apache website. Well there is another compatibility hick-up between WebLogic 9.2.1 and Apache Httpd latest versions with the proxy plug-in provided. So I had to pull back to Apache Httpd 2.0.59.
Since the plan was to use two layers (web layer and application layer) distributed on multiple machines in a heterogeneous environment, three machines were considered with the following configuration.

• First machine with an IP address 1.1.1.1 running Redhat Linux 5.2.1 on it, referred as WL1 for the rest of this article.
• Second machine with an IP address 1.1.1.2 running Windows2003 on it (referred as WL2).
• Third machine with an IP address 1.1.1.3 running WindowsXP on it (referred as AP1).

While selecting a load balancer I had two options. One is to use the hardware load balancer and the other is using the application server proxy plug-in. Since the goal is to avoid the former one, the later was opted for. WebLogic provides a proxy plug-in for most of the popular web servers to be used between the web server and application server layers. It helps the web server to balance the load and route the requests to the appropriate WebLogic server in a cluster. This plug-in can be configured to support SSL as well, but I decided to avoid that for now.

As per my plan I sketched a topology as shown in figure 1 below.

Figure 1

Installation of WebLogic on WL1

To proceed with the installation of WebLogic on WL1, I followed the steps as below.

· Created a directory as /opt/bea.
· Copied the platform921_linux32.bin to /opt/bea.
· Used “chmod a+x platform921_linux32.bin ./platform921_linux32.bin” to change it’s mode to execute.
· Used “./platform921_linux32.bin -mode=console -log=./platform_install.log” to install in console mode. For a detailed description on how to start WebLogic in console mode follow the instructions given at Starting the Installation Program.
· For a step by step instruction on how to continue with the installation program you can refer to the article Running the Installation Program in Console Mode.
· The BEA_HOME was made to point to /opt/bea.
· Installed only the “Server” and “Web Server Plug-Ins” from the list.
· Selected the server startup more as “Production Mode”, ignoring the installation of “Mercury profiling tools”. This is installed only for “Development Mode”.
· The product directory in my case was /opt/bea/weblogic92.
· Once the installation script was completed, to set the environment I executed “./setWLSEnv.sh” under /opt/bea/weblogic92/server/bin.

Installation of WebLogic on WL2

Now we need to switch over to WL2 machine (IP: 1.1.1.2 running Window 2003) to install WebLogic. For this I
· Copied the platform921_win32.exe on WL2.
· Created a directory c:\bea as the BEA_HOME.
· Started platform921_win32.exe.
· The rest of the steps are pretty much similar as followed for WL1 except the command line interface, rather you get a cool and nice GUI mode for installation on windows.

Once we are finished installing WebLogic on WL2, a windows service will be automatically installed and started for WebLogic Node Manager. Node Manager is a software agent employed by WebLogic to control server instances remotely.

Configuring WebLogic Admin and Domain

As the most vital part, we will configure the WebLogic Admin Server and domain on WL01. Here are the steps I followed.
· Moved in to /opt/bea/weblogic92/common/bin on WL1.
· Started “./config.sh –mode=console” to start the Configuration Wizard in console mode. You can refer to the article - using Configuration Wizard for details.
· Picked up “Create a new WebLogic domain” in the “Welcome” screen.
· Selected “Choose WebLogic Platform Components” in “Select Domain Source” screen.
· Entered WebLogic Admin Server username and password both as “weblogic”.
· In “Domain Mode Configuration” screen “Production Mode” is selected.
· In “Java SDK Selection” screen “Sun JDK 1.5.0_06” is selected.
· In “Choose Configuration Option” screen “Yes” is selected.
· Modified the Admin Server name as “MyAdmin”.
· Modified the IP address to 1.1.1.1. By default the IP address points to 127.0.0.1 and there is a problem with using an IP address of x.0.0.1 range as your multicast IP (refer to Using WebLogic Server Clusters). Finally the "Choose Configuration Option" screen looked like below.

Name Value
__________________________________
1 *Name: MyAdmin
2 Listen address: 1.1.1.1
3 Listen port: 7001
4 SSL listen port: N/A
5 SSL enabled: false

I skipped all steps to reach the “Edit Domain Information” screen and entered the domain name as MyDomain. Once the config.sh script gets completed on WL1 we can see the domain is created under /opt/bea/user_projects/domains/MyDomain unless you have specified a different path for the domain directory during the configuration.

So far WebLogic is installed on WL1 and WL2. The Administration Server is configured on WL1. Since I am slow at typing, carrying out the rest of the process with the Admin Console GUI was an easy option for me. Just a few steps before that to start the Admin Server
· Added ReverseDnsEnabled=true in /opt/bea/weblogic92/common/nodemanager/nodemanager.properties. Same change was made on WL2 as well. This helps the Node Manager to lookup hostnames through DNS entries both on 1.1.1.1 and 1.1.1.2.
· Moved over to /opt/bea/weblogic92/server/bin and started the Node Manager as a background process by “./startNodeManager.sh &”.
· Moved in to /opt/bea/user_projects/domains/MyDomain. Added the lines below in startWebLogic.sh to avoid the username and password prompt and then save the file.

WLS_USER=”weblogic”
export WLS_USER
WLS_PW=”weblogic”

export WLS_PW

Figure 2

Starting Admin Server on WL1

· Start the Admin Server by “./startWebLogic.sh”. This will start the Admin Server.
· After the server has been started you can open the Admin console GUI on your browser at http://1.1.1.1:7001/console and log in using the username and password that you have specified during the Admin Server configuration. I used “weblogic” as username and password both.
· Once the Admin Console opens up, select the MyDomain -> Environment -> Servers to see if the Admin Server is running with good health as shown below in figure 2.

Creating WebLogic Cluster

· To create a cluster, follow the MyDomain -> Environment -> Clusters link. Click the “Lock & Edit” button on the top left corner to be able to edit the configuration page.On the right pane, click on the “New” button and specify the cluster details in the page as shown in figure 3. A WebLogic domain can be constituted of multiple clusters. Well I need just one for now.

Figure 3

The multicast address is set as 239.192.0.0 and the port is set as 7001 by default. I altered them to ensure that there is no conflict with another processes already running on the same machine and port. You can set the address to any IP between 224.0.0.0 and 239.255.255.255 (refer to Using WebLogic Server Clusters).
WebLogic Server uses IP multicast communication among the servers in a cluster mainly for two purposes.
1. To send and receive Cluster-wide JNDI updates. Since WebLogic uses a Shared Global JNDI implementation, the servers keep on monitoring the availability of the objects in the local JNDI tree and update other members of the cluster about it. Similarly they receive messages from other members and update their own JNDI tree accordingly.
2. To send and receive Heartbeat messages. This communication helps a member server in a cluster to determine whether another member has failed.
Click on the “Activate Changes” button at the top left corner to send a request to the Admin Server to actually create the cluster (figure 4). The cluster does not have any servers in it till now. We will add servers to it in the following steps.

Figure 4

Creating WebLogic Machines

In the following step you need to create and configure WebLogic Machines through the "Configure Machines" screen. A WebLogic Machine is a logical representation of the physical machine (computer) that hosts one or more WebLogic Server instances. This is specially required when the cluster spans across different physical machines. There are two main purposes to have WebLogic Machines.
1. To ensure that server session data is replicated on separate pieces of hardware.
2. To ensure the Node Manager is able to have control on the remote servers.

If a Node Manager runs on a machine that does not host an Administration Server, as in the case of my WL2 machine, so I created two machine configurations for WL1 and WL2.
· Follow the link MyDomain -> Environment -> Machines and click on “Lock & Edit” button.
· Click “New’ button on right and enter the machine name as “MyMachine01’ in the following screen.
· Select the “Machine OS” as “other” and click “OK”.
· Click on “Activate Changes” button on the next page at the top left corner.Once the machine is created as MyMachine01, follow the link on “MyMachine01” and then to “Configuration -> Node Manager” tab to arrive at the page as shown in figure 5.

Figure 5

• The “Listen Address” was changed to 1.1.1.1 and followed by a click on the “Save” button.
• You need to click to “Activate Changes” button at this point.
• This will modify the Node Manager address when it’s I restarted next time.
• Similarly two WebLogic machines were created as described below.

  Name*       Node manager listen address Node manager listen port
__________________________________________________________________
1 MyMachine01                     1.1.1.1                     5556
2 MyMachine02                     1.1.1.2                     5556

Creating Managed Servers

• Next part is to key in the details for the managed servers.
• Follow the link MyDomain -> Environment -> Servers and click on the “Lock & Edit” button again.
• Click on “New” button on the top of the servers table on the right.
• Enter the details as shown in figure 6 to create a managed server called “MyManaged01” in MyCluster01 and click “Next” button. Click “Finish” button in the following screen.

Figure 6

• Once the managed server is created as MyManaged01 in the server table, I followed the link on “MyManaged01’ to go to the Configuration -> General tab in “Settings for MyManaged01” page and clicked on “Lock & Edit” button.
• You need to modify the “machine” to “MyMachine01” here to assign this server to MyMachine01. Figure 7 shows the configuration change.
• To apply the change we need to click on “Save” button and then on the “Activate Changes” button.

Figure 7

Similarly four managed servers were created as described in the table below.

    Name*         Listen address  Listen port  SSL listen port  SSL enabled
___________________________________________________________________________
  1 MyManaged01   1.1.1.1         7101               N/A        false
  2 MyManaged02   1.1.1.1         7201               N/A        false
  3 MyManaged03   1.1.1.2         7301               N/A        false
  4 MyManaged04   1.1.1.2         7401               N/A        false

Notice that, I have created MyAdmin as the WebLogic Admin Server listening on port 7001, two managed servers as MyManaged01 and MyManaged02 listening on ports 7101 and 7201 respectively on WL1 machine (IP: 1.1.1.1). We need to ensure that the combination of IP and port should be unique for all the server processes. Rest of the two managed servers MyManaged03 and MyManaged04 were created on machine WL2 (IP: 1.1.1.2) and made to listen on ports 7301 and 7401 respectively. When all managed servers were created, the server table appeared as shown in figure 8.

Figure 8

Configuring Node Manager on WL2 using WLST

As per my strategy, the Administration Server is going to be on WL1 and WL2 machine will run a Node Manager to communicate with the Administration Server for controlling the cluster. To accomplish this, the Node Manager on WL2 has to be registered with the Admin Server hosted on WL1. I did it with the aid of WebLogic scripting tool (WLST). The WebLogic Scripting Tool (WLST) is a command-line scripting interface that system administrators and operators use to monitor and manage WebLogic Server instances and domains. The WLST scripting environment is based on the Java scripting interpreter, Jython. The steps are given below.

• Log in to WL2 machine.
• Open a command window and move in to C:\bea\weblogic92\common\bin\ directory.
• Start wlst.cmd. It takes you to a prompt like wlst:/offline>
• Issue the command below at the prompt. It connects WLST to the Admin Server running on WL1.
• wlst:/offline>connect('weblogic', “weblogic', 't3://1.1.1.1:7001')
• The prompt immediately changes to wls:/MyDomain/serverConfig> if the connection is successful.
• To register "MyDomain" with the node manager running onWL2 use
  nmEnroll("c:\\bea\\user_projects\\domains\\MyDomain"). Directories will be created by node manager automatically after doing this.
• Come out of WLST with wls:/MyDomain/serverConfig>exit().
• Restart the Node manager (Control Panel -> Admin. Tools->Services) on WL2.

Figure 9

Starting the Cluster

Finally we can now start our cluster from the Admin Console (http://1.1.1.1:7001/console).

• Go to the MyDomain and on the right pane go to the “Control” tab.
• Check the boxes for all four managed servers and hit “Start” button as depicted in figure 9.
• I came across an error while starting my servers as

<BEA-090482> <BAD_CERTIFICATE alert was received from 1.1.1.1

I found out a remedy to avoid this error but still looking for a permanent workaround. To fix this, click on the “Lock & Edit’” button to start editing the configuration. Go to MyDomain -> Environment -> Servers -> MyAdmin -> Configuration -> SSL -> Advanced and turn the “Hostname Verification” to “None”. Save the changes by clicking the “Save” and then the “Activate Changes” buttons. You should restart the Admin Server at this stage to apply the changes. Once the servers are started the server table will look like as shown in figure 10.

Figure 10

Keep refreshing this page and it will finally come up with a status as shown in figure 11 indicating that your WebLogic platform is up and running with a multihomed (with multiple machines) clustered topology.

Figure 11

Deploying and starting MyWeb application on WebLogic cluster

Our following step will be to deploy the “MyWeb” application on “MyCluster01”.

• Before starting the deployment a useful step is to enable the Clustered Constraints. It enables the two phase deployment in a clustered environment. This feature enables applications to be deployed successfully on all the member servers at the same time. If one server fails the entire process is stopped. You can go to MyDomain -> Configuration -> General tab and turn on the “Enable Clustered Constraints“.
• I created some JDBC data sources since MyWeb needs a couple of them. You can create JDBC data sources under MyDomain -> Services -> JDBC -> Data Sources.

MyWeb application was packed in a WAR file. Before packing you application ensure that the WEB-INF directory should have a weblogic.xml containing the lines below.

<session-descriptor>
<session-param>
<param-name>PersistentStoreType</param-name>
<param-value>replicated</param-value>
</session-param>
</session-descriptor>

The above lines ensure the replication of Http session for an application.

• To deploy the application, MyWeb.war was copied under a temporary directory /opt/bea/temp/apps/.
• The mode of the file was changed as chmod 644 MyWeb.war.
• On the Admin Console, you can follow the link to MyDomain -> Deployments and click on the “Install” button on the Deployments table to begin deployment. The next page shows a link to the root directory of WL1. If you don’t see the install button as active you must have forgotten a click on the “Lock & Edit” button.
• Follow the link to 1.1.1.1/opt/bea/apps/ where MyWeb.war file is kept.
• Pick up MyWeb.war to deploy. You have to click on the “Next” button to proceed from here.
  

Figure 12

• In the next page select “Install this deployment as an application” and click “Next”.
• The following page will ask for a target for the deployment where you need to select MyCluster01. With this option the application will be deployed on all the servers belonging to MyCluster01. Click “Next” here.The following page will appear as shown in figure 12. I accepted the default option shown in this page and clicked next. The default source accessibility option i.e. “Use the defaults defined by the deployment's targets” means the deployment will be done in a staged mode for all the managed servers and non-staged mode for the Admin server (not applicable in my case since I am not deploying MyWeb on the Admin server). In a staged deployment the source WAR file will be downloaded or copied by all the servers to have their own copies, while the non-staged option shares the source from a single location. Staged mode is specially suggested when you have a multihomed WebLogic domain. It avoids the single point of failure. Supposing in a non-staged mode, when the server hosting the application source goes down the source will be unavailable to other servers in the cluster. I clicked the “Finish” at this page. The next page appeared as shown in figure 13.

Figure 13

• The status of MyWeb application will appear as “distribute Initializing” Click “Activate Changes” button here. The status will change to “Prepared” immediately.
• The final step is to start the application. Click on the check box on the left of MyWeb and then on the “Start” button drop down. Select “Servicing all requests” option from the “Start” button drop down.
• Click the “Yes” button on the following page and Deployment table will appear as in figure 14 indicating that MyWeb application has been started successfully.

Figure 14

With this you can conclude the WebLogic configuration chapter. Well that’s not all.

Installing and configuring Apache Httpd on AP1

I still need to setup my web server layer on the machine AP1 (IP: 1.1.1.3).

• To proceed with that start apache_2.0.59-win32-x86-no_ssl.msi to install Apache Httpd 2.0.59 on AP1 under C:\Dev\apache\httpd\Apache2.
• Once Apache Httpd is installed the Apache Service Monitor will be started.
• Since one of the goals of my R&D was to separate out the static and dynamic contents among web and application server layers, I created a directory as C:\Dev\apache\httpd\Apache2\htdocs\MyWeb on AP1 and placed the static contents of MyWeb application e.g. HTMLs, Java-scripts, images and style-sheets under this folder. Notice that you should create the directory with same name as specified for the application context in WebLogic (shown in the deployment table in figure 13).

Adding WebLogic Proxy Plug-in on Apache

• To install the WebLogic proxy plug-in for Apache Httpd, I copied the mod_wl_20.so file from /opt/bea/weblogic92/server/plugin/win/32/ on WL1 to C:\Dev\apache\httpd\Apache2\modules\ on AP1. mod_wl_20.so is the proxy plug-in provided by WebLogic for Apache Httpd 2.0.59.
• More information on Installing and Configuring the Apache HTTP Server Plug-In can be found on WebLogic website.
• To configure the proxy plug-in it’s mandatory that the mod_so.c module is enabled. I went to C:\Dev\apache\httpd\Apache2\bin and executed apache –l to verify that. It shows a list of modules installed with Apache.To make Apache point to the WebLogic cluster using the proxy plug-in, add the lines below in the C:\Dev\apache\httpd\Apache2\conf\httpd.conf file. The lines instruct Apache to route all requests demanding for dynamic contents to WebLogic. Since MyWeb was using Struts I added *.do extension as a parameter to MatchExpression option. The wlproxy.log is configured to record the activities during the routing. Logging and debugging parameters are optional.

LoadModule weblogic_module modules/mod_wl_20.so

<IfModule mod_weblogic.c>
WebLogicCluster 1.1.1.1:7101, 1.1.1.1:7201, 1.1.1.2:7301, 1.1.1.2:7401 MatchExpression *.jsp
MatchExpression *.do
Debug ON
WLLogFile c:/dev/apache/httpd/apache2/logs/wlproxy.log
DebugConfigInfo On
</IfModule>

• To route all request irrespective of static or dynamic content you can add the following lines.

<Location /MyWeb >SetHandler weblogic-handler</Location>

This will route all requests containing /MyWeb to the WebLogic cluster.

Testing failover and session replication with MyWeb

In my case the welcome file was MyHtml.html, so I brought up MyWeb at http://1.1.1.3/MyWeb/MyHtml.html. If you have an index.html to start with you can simply type http://1.1.1.3/MyWeb as the URL. MyWeb appeared as shown in figure 15 on my browser. See how it works with you.
You can check the c:/dev/apache/httpd/apache2/logs/wlproxy.log file to find out the name of the WebLogic managed instance to which the request is actually getting routed. From the log I discovered that MyManaged01 on WL1 was receiving the requests.
I checked the /opt/bea/user_projects/domains/MyDomain/servers/MyManaged01/logs/access.log to confirm that it’s really receiving the requests.

• I started a session with MyWeb and played around a little bit with it.
• Tried shutting down MyManaged01 forcibly from the WebLogic Admin Console (MyDomain -> Control tab) and found that the session got replicated to MyManaged02 on WL1. I further tried getting MyManaged02 down and found the session to be replicated to MyManaged03 on WL2.

Figure 15

Notes

Socket Exception

There are couple of points that I would like to mention here. Since MyWeb had a large size of nearly 250mb I faced an error during the staged deployment as mentioned below.

<DeploymentService> <BEA-290065> <Deployment service servlet encountered an Exception while handling the deployment datatransfer message for request id "1,190,838,152,600" from server "MyManaged03". Exception is: "java.net.SocketException: Socket closed.
……
……..
java.io.IOException: Reached EOF
……
……..
java.lang.IllegalStateException: Response already committed <Error> <HTTP> <BEA-101083> <Connection failure.
……
……..
java.net.SocketException: Error in poll for fd=52, revents=32
……
……..
<Deployer> <BEA-149078> <Stack trace for message 149004
weblogic.management.DeploymentException: Exception occured while downloading files
……
……..

After having a tough time while searching for an actual reason for the errors, I found that during the staged deployment WebLogic uses an Http Servlet to download the application source to each of the managed servers in the cluster. The servlet was getting timed out and closing connections to other members in the cluster. As a remedy I followed the steps below.

• Set the CompleteMessageTimeout in MyDomain -> Environment -> Servers--><server>-->protocol-->general tab from 60 seconds to 480 seconds for both the admin and all managed servers. Where “<server>” indicates the name of the server.
• Set the IdleConnectionTimeout to 480 seconds for the admin and managed servers.
• Went to Mydomain -> Environment -> Servers--><server>-->protocol-->HTTP and change PostTimeout to 120 (default:30 secs).

Try the same if you come across a similar problem.

Replication groups

During this experiment I tried building replication groups. A replication group is a preferred list of clustered servers to be used for storing session state replicas. WebLogic uses them to support failover in a cluster. WebLogic Server attempts to create session state replicas on a different machine than the one that hosts the primary session state. You can further control where secondary states are placed using replication groups.
In my case I created two replication groups as RepGroup13 and RepGroup24. RepGroup13 consisted of MyManaged01 and MyManaged03, while MyManaged02 and MyManaged04 were members of RepGroup24 as shown in figure 16. RepGroup13 was made to be the preferred secondary replication group for the members of RepGroup24; similarly RepGroup24 was designated as the preferred secondary group for the members of RepGroup13.
Considering all managed servers are up and running, once MyManaged01 receives a request and starts a session with a client, the session will be replicated preferably on MyManaged04 as a secondary server (based on the server ranking policy) rather than randomizing on any other member. Incase MyManaged01 goes down MyManaged04 will continue to serve the client with the session state collected from the point where MyManaged01 has collapsed. If replication groups are not configured WebLogic will pick up any member in the cluster as the secondary target based on a round-robin policy.

Figure 16

I followed the steps below to create replication groups.

• Opened Admin Console follow MyDomain -> Environment -> Servers -> MyManaged01 -> Configuration -> Cluster.
• Entered RepGroup13 as the Replication Group for MyManaged01.
• Entered RepGroup24 as the Preferred Secondary Group for MyManaged01 as shown in figure 17.
• Saved Configuration by “Save” and “Activate Changes” buttons.
• Similarly set RepGroup13 as the Replication Group and RepGroup24 as the Secondary Replication Group for MyManaged03.
• Set RepGroup24 as the Replication Group and set RepGroup13 as the Preferred Secondary Group for MyManage02 and MyManage04.
• Restarted WebLogic server.
  

Figure 17

My next plan is to try a similar R&D on WebSphere and Apache Httpd unless I get caught with something more urgent, till then keep clustering ;)