BAM: Capabilities & Applicability (Part 2)

BAM Architecture in General

Architectural diagram on figure below shows general architecture of BAM solution and explain functions of each of them.


Modeling tool

Usually GUI based toolkit is used to define monitor model. The model contains definition of events that should be “caught”, metrics, triggers and KPIs. Finished model is deployed and monitoring can start.


BAM core

This is the main component of BAM solution. It generates event patterns according to the monitor model and deploys them on CEP (Complex Event Processing) engine. It later processes the received event data and transforms them in the desired form. BAM core can also communicate with the Process engine, start and stop the processes, generate events etc. according to triggers and alarms defined in the monitor model.


CEP engine

This is the lowest level component responsible for event filtering. The CEP engine receives query patterns from the BAM core. It queries the event flow both from the Process engine and ESB and sends the “wanted” events back to BAM core. Usually only process engine events are used for BAM, but CEP engine allows also filtering of ESB flow events. That can be used in more technical oriented monitoring, or combined with the business data, to monitor performance, resources usage etc.


Front-end

Front-end receives data already processed, with computed KPIs etc. and takes care of their visualization. Every BAM solution should deliver graphical front-end witch rich visualization possibilities, such as charts, dashboards or graphs. An advanced OLAP-like visualization tool can be also useful. One of the advanced features some front-ends also provide is a real-time view of processes, visualized on process graph. Here you can view process diagram as it was modeled and track the state of a particular process instance. It is also common that at least one front-end should be web based. In this case front-end should provide a possibility to assemble your dashboards “on fly”. This is done usually by AJAX based scripts. AJAX becomes a standard for web-based BI tools today and it is also supported by Jasper Dashboards, one of the hot candidates for open source BAM front-end.


Long term data mining BI tools

An optional component that can analyze data from long-term perspective, realize data mining and knowledge discovery processes. Discovered information is delivered back to BAM core. Mostly external tools are used for such cases nowadays but hopefully in future some of those tools will become a part of BAM solutions. That will allow the feedback obtained by those tools to be sent back to the bam core or the process server.

General architecture for BAM

Where to Apply BAM

A business process typically spreads across multiple large systems which in turn made up of subsystems and components. For example, an order (i.e. a business entity) for a broadband connection flows from one system to another. It goes through some level of transformation while getting processed. The figure 1 shows how an order flows among multiple systems such as CRM, Billing and Inventory etc. to fulfill an order in a typical business process.
Once the order for the broadband connection is placed by the customer online, it flows through different systems on the way to get fulfilled. The basic steps followed in the process above are:

• The order must be validated online for availability and geographical access.
• A broadband router must be shipped from the warehouse to the customer’s address.
• Customer’s phone line and broadband connection must be linked to the same billing account.
• CRM should be updated with the order fulfillment information for the call center to access when providing support to the customer.
The fault with this setup is that each of the systems is autonomous. They have their own alerting and reporting mechanism. This might lead to a scenario where the customer experiences problems with - the delivery of the broadband router in time, unable to receive a discount (if associated) on the combination of a broadband and a land line. Due to the absence of a proper BAM solution the service provider won’t be able to sense these faults before the customers do, causing an unwanted blot in the customer relationship.

In an opposite scenario with a BAM implemented on the service provider’s side:


• The data from not only the systems involved in the business process but also the related subsystems and components can be captured proactively.
• The collected data can be tracked and analyzed against the defined measures.
• Alerts can be raised to the applications and the people when something is not behaving as desired.

How to Employ BAM

To be able to employ a BAM solution effectively the preferred option is to put the entire business process under the supervision of a BPM engine as shown in the figure below.

Business processes executed by BPM

 

This setup can be extended to incorporate a BAM solution as shown in figure below. Under this situation, the BAM adapter is integrated with the BPM engine that in turn is integrated with all the business process systems. The BAM adapter traces the business entity through the business process and keeps comparing it against the standard or defined values. Let us set a KPI for efficient delivery in the abovementioned example, such that an order should not rest within the inventory system more than seven hours. To handle this situation the BAM adapter should track all the orders that go to the inventory management system. If it does not receive a status change notification within seven hours, an email or an SMS alert is sent to the system owner. In case an API (application programming interface) is available to interact with the inventory system, the specific order can be identified and the owner can be notified with an alert on the same. If there is no status change noticed within twenty four hours, an alert (email or SMS) can be sent to the customer with an updated date of delivery.

BAM integrated with BPM infrastructure

This way, BAM brings in agility and resilience in the business, not only that, it can also cut down the burden on the support services by proactively notifying the customers about their order status.

Enterprises that use middleware instead of a BPM infrastructure for business process orchestration can also incorporate the BAM support in their existing setup using an Enterprise Service Bus (ESB) and a logging framework.

In this case, the applications that cater to the part of the business process will require using a BAM logging framework for updating the status of the business entity (e.g. order) through its lifecycle. Multiple instances of this logging framework can be deployed throughout the enterprise and all of them can forward the logging calls to the BAM engine through the use of an ESB or an asynchronous messaging service.

BAM incorporated in a setup with middleware but no BPM

The BAM engine can revive the lifecycle trail of the business entity from these logged messages, trace its progress and raise alerts, alarms and communications as appropriate.

BAM: Capabilities & Applicability (Part 1)

Nowadays BAM (Business activity monitoring) is catching up like a mandate for the administration and management for an enterprise. There are few topics that discussed in this article explaining the capability and applicability of a BAM system. The references to the telecom IT sector has been used due to my current involvement in telecom industry.

Proactive data mining

Like other industry verticals the telecom sector depends on the execution of complex business processes. Usually business processes include interactions among multiple heterogeneous IT systems and human resources. The efficiency and success of these business processes in an operational environment can be monitored based on multiple Key Performance Indicators (KPIs) of the business such as customer satisfaction, overall business performance and adaptability to changes. As a reason it is becoming important to monitor the performance of the business processes and optimize them based on the results. Presently businesses come to know about a problem when a failure occurs in the business processes. Monitoring is one of the best ways to sense an imminent problem and avoid it with corrective actions in advance so that, the impact on business and the users can be minimized.
This article will try to present a solution to monitor and optimize the business processes based on the currently available technologies and their maturity.

Purpose of Business Activity Monitoring

The term business activity monitoring was introduced by Gartner, Inc. in 2001 as their notion of aggregation, analysis, and real-time presentation functionality of business relevant information. The main objective of business activity monitoring is to predict or sense the upcoming problem based on the operational transactions and report the same for a preventive measure.

It is becoming increasingly significant in today’s enterprises to consider informed decisions in timely manner by consolidating information present in various silo data sources. These decisions eventually have major impact on the business parameters such as profitability, revenue, customer satisfaction and agility. IT systems like Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), Order Management System (OMS), Business Process Management (BPM), etc. have served with considerable efficiencies in their own areas of the business. However mostly the business processes span across multiple systems.
If we imagine of an enterprise with the capability of foreseeing the next steps, predicting impact of events, sensing an unseen fault by putting two and two together from different systems and eventually spot and plug any hole in the plumbing before the damage is done, a tremendous agility, resilience and efficiency in the business processes can be introduced.
A layer that is able to monitor business processes in the above mentioned manner should be able to understand and perform real time monitoring of vital signs in the context of business process. This monitoring should be made proactive as much as possible. Apart from the failures, the deviations in the normal trends based on the historical data need to be flagged and alerted to appropriate stakeholders with minimum time lag. Among the on-demand features, it should have ad hoc reports, dashboards for the decision making community within the enterprise. Such a layer would bring in manifold benefits.
For example, let’s consider an e-commerce scenario with the key business process of fulfillment of orders placed online. The customer places an order with a set of items and gets notified with a date of delivery. Under a happy path scenario, the customer receives the items on the promised date. Considering a variation when the logistics of delivery such as trucks, delivery boys etc. are operating at the full capacity due to a sudden increase in the number orders placed in, an individual order might get delayed in the delivery. The activity monitoring layer could potentially correlate the high volumes being handled by the system to a potential delay in delivery times. This can trigger an alert or communication to customers informing them about the delay before it actually takes place.
Imagine a more interesting scenario of inventory management for the e-commerce site. Assuming that there is a forecast for thunderstorms for couple of days in some of the areas, the monitoring layer can send an alert to the inventory management group to stock up on rainy weather essentials such as umbrellas and raincoats to enable the business to meet the un-foreseen increased demand.


Components of a Business Activity Monitoring

BAM consists of the following generic components:

Business Event

A business event points to the occurrence of an event that is significant to an operation of a business activity. It is separate from an IT event. An IT event usually shows the occurrence of an event in the context of an IT resource.

Business activity

A business activity offers services or functionalities required to serve a specific business purpose.

Metric

A metric is a measurement of a specific property belonging to business activity that can be used to monitor business operations.

Key performance indicator

KPIs are the metrics used to measure the progress of a business goal. KPIs have to be measurable as well as achievable and provide the possibility to aggregate measures over a set of instances. In order to define KPIs, companies should have clear performance requirements, and they must also have a clear definition of their strategic target which can be compared to runtime values.

Techniques of Today

Though BAM is relatively a new term are some techniques that have been around for a while. The stalwarts amongst these are

• IT system monitors
• Business Intelligence (BI)
• Reporting and Analytics
• Business Process Management (BPM)

Each of the above has its own pros and cons when the BAM features are evaluated. Let us dig a little deeper in these three techniques.


IT System Monitors: To be able to monitor failures and success events at times many enterprises have implemented IT system monitoring measures in effective ways. The main objective of such systems is to alert the support services in a timely manner to meet the service level of agreement and availability levels of the application. This technique targets individual applications for monitoring and helps them to react under a error condition at the application level. While it does not consider the entire business process as a monitoring target as a result it is unable to track any fault at the business process level. IT system monitors are mostly reactive than proactive. They keep watching the applications, logs, network, server, hardware in isolation for faults rather than the end-to-end business process. IT system monitors will be unable to detect an error at the business process level while the applications are running smoothly. IT system monitors watch for the malfunctions and failures only, while an ideal BAM should look for success and failures both to yield meaningful activity metrics. They are also not aware of the business process level KPIs (Key Performance Indicators).
For example, if a customer is promised with the certain duration for an order fulfillment process, a KPI needs to be set at the business process level for order processing. IT system monitors are unable to detect a degradation of this kind of KPIs. The scope of alerts is limited to the application and does not correlate to failures in other participating application in the same process. It is also unable to detect any variation in the business process when the applications are functioning properly.
In spite of all shortcomings these tools have been serving as monitoring instruments for many organizations. In fact there have been conscious design efforts introduced to log success events (along with the failures) in individual applications’ logs against a unique correlation id and also a listener mechanism to poll the logs in a timely manner and correlate the events using the unique id for each business process transactions. However since the number of I/O operations increase (due to write in log and timely polling of listeners), the performance of the applications is compromised seriously under this situation.


Business Intelligence (BI): Business Intelligence has been occupying the place of a backbone for decision making process of the organizations for quite some time now. This is due to its capabilities of analytics and extensive reporting. BI is capable of processing enormous amount of data stored in huge data marts at predefined frequencies by using ETL (Extraction Transformation and Loading) tools and put the same into the data stores which gets reported eventually. This way BI enables businesses to make informed decisions to a great extent. However when it comes to BAM some constraints apply.

• Since BI and data warehousing are based on batch processes, proactive and real time decisions can not be made easily using BI.
• The statistical analysis is performed manually in BI. Since the monitoring framework is not updated with the historical trends, it’s not able to flag events which are deviating from the normal trend.
• Since BI is not a real time system, it can not track real time errors or deviations and generate alerts for that.

Business Process Management (BPM): Enterprises use business process management to manage, model and automate business processes. Externalization of all business processes those are entrenched deeply in applications to a central location is one of the main tenets of using BPM. BPM accomplishes quite a lot from that angle in terms of improved process agility and centralized governance of vital processes. But BPM also has some shortcomings as a BAM solution:

• BPM has no built in monitoring capabilities. Therefore, alerts based on the success or failure of a process can not be generated by the BPM infrastructure. Furthermore the success and failures can not be evaluated against the historical trend to compose intelligent reports necessary for the right changes or decisions to the activities.
• BPM has the knowledge of a business process while it is lacking the knowledge of relationship between the participating applications, historical data and statistical analysis which are required for anticipating the unknown errors in a business process.

The table below presents a comparative analysis of the BAM techniques described above. It also shows the rating of the three techniques as an ideal BAM solution.

Table 1: Comparative analysis of three BAM techniques