Start seeing with SharePath free download here

Have any questions? Just call us 508-318-6488

January 29, 2014

No Comments

insurance-topology

The Transaction Tracing Opportunity

Corporate computing environments can be complex involving many different programming languages, applications, middleware components, and endpoints. Many composite applications integrate new software, legacy code, and packaged solutions. Enterprise application performance monitoring works by solving the transaction tracing problem.  This means finding a way to identity application latency by tracking a transaction from a mobile application or browser (or a Windows desktop), across the application server and messaging system, to one or more databases and then back to the user. (And that is just a simple example.)

This is called the “holistic” approach to performance monitoring.  What matters to the end user is not the seek time on the disk, the latency across the network, or the SQL statement running out of control.  The user just wants to know why the application is running sloooooow.

This application performance monitoring tool knows the architecture of the entire IT infrastructure, so it knows what components are involved in each type of transaction.  The monitoring tool presents a dashboard to the IT operations staff, so they can monitor end-to-end performance.  If the application is operating within service levels then everything is green.  If not, the dashboard turns red.  Either way the tools lets the analyst click on components in the topology and drill down to see the response time of each piece.

The performance monitoring tool presents two possibilities to the analyst:  (1) the dashboard is red or (2) the dashboard is green.

Dashboard is Red

If the help desk starts getting calls that the system is slow, the support persons consults the application monitoring dashboard.  If the dashboard shows red then the application is operating outside norms or established thresholds.  With the end-to-end performance monitoring tool, the analyst selects the application marked red and then clicks to show the tiers or components that make up the transaction.  The tool breaks down the application into each tier–LDAP, shared services, web server, proxy, messaging, and application server–showing the response time in each.  Then the analyst clicks on any of the components, the performance monitoring tool shows the response time for each transaction marking in red those components that are operating outside norms.

Dashboard is Green

What do you do when users calls and the dashboard is green?  Green means the application is operating within the agreed service level, say, 99%.  But that number is only an average; it does not include all events.

If the user is experiencing latency and the dashboard is green, the analyst cannot infuriate the customer by telling them that everything is green and he or she cannot reproduce the problem.  Instead the analyst needs to drill down into the application to get transaction-level details.  If the top level dash board shows green, drill down into the application the user is using.  Then drill down into each component until you get transaction-level details.  Sort these by response time.  There the analyst sees that a particular transaction is taking too long.  Click on the transaction to see the details of, for example, the Java methods or the LDAP calls. It could be that this user is looking up something or entering data that is not so frequently used.  Then analyst could discover, for example, that the LDAP group lookup (&(cn=something)(objectclass=group)) is taking too long.  This could indicate a corruption the in the LDAP for that particular group or a coding issue.

This is the basic approach to using the performance monitoring tool to solve the transaction tracing problem.

November 28, 2011

No Comments

Monitoring vs. HR

In my many years of managing data centers and IT, I’ve had (and still have) the distinct honor to work with a diverse group of people – developers, SYS admins, DBA’s, first, second, and third level support. What I have realized during those years is that, luckily, there isn’t much difference between IT and people.

People’s lives are constantly changing, just like IT. There are several situations which occur unexpectedly in one’s life; trouble with the mortgage, kids needing braces, a losing basketball team, or even just a lousy mood can skew a person’s outlook. Even positive things cause changes: a great movie, relaxing vacation, or desire for promotion brings about a new state of mind. No matter what has changed, it leads to an impact on the job. The smallest difference in a worker’s life, for better or worse, can change productivity for an entire day, week, month, or year.

Just like IT, it’s very important to monitor any and all changes, since even the slightest difference could have a “butterfly effect,” i.e., a causal chain that leads to major upsets down the line. End-to-end monitoring of your people is in-depth. You need to listen very carefully and ask them the right questions. By building relationships, and interacting non-stop, over and over again, you can at least understand, respond, and manage the changes that happen outside of your control.

An old professor from my MBA studies left me with some knowledge that I’ll never forget. He said “your employees will always talk, always complain. That’s a fact and you can’t change it. The challenge is to get them to talk to you.”

IT components will always change, that’s the nature of a “good” IT environment. Some of them will cause huge headaches for IT management and that’s a fact we must live with. The challenge is to make those changes visible, from end to end, so that we can effectively monitor and react to them. I wish I had a performance management tool for Human Resources! Now, I’m just doing it manually!

September 27, 2011

No Comments

End-to-End Monitoring of Transactions

End-to-end monitoring solutions should monitor all of the infrastructure’s components, yet some solutions only monitor one part of the bigger picture.

The term “end-to-end” relative to transaction monitoring is very over-used. End-to-end Website monitoring, end-to-end server performance monitoring, end-to-end database monitoring, and end-to-end network performance monitoring. The list can go on and on when, in fact, they only provide monitoring for one small part of the bigger picture, as illustrated below.

What End to End Means to different solutions

Why True End-to-End Transaction Monitoring is Different than Monitoring Server Performance

Monitoring server performance traditionally refers to ensuring that CPU utilization and memory consumption have not reached maximum levels. However, this does not promise true application availability. Traditional end-to-end monitoring tools provide a dashboard that shows the health of each one of the individual components within the data center. But what if someone forgot to re-connect a network cable after maintenance or what if a load balancer was configured incorrectly and is not executing a proper round robin? An alert will indicate that something is wrong with the servers that are still connected when, in fact, the problem is that traffic is simply not being equally distributed. A network performance monitoring solution could be implemented, which would aid by monitoring the usage of the network, but that is not a single end-to-end solution anymore.

What if a runaway process in one server is “bombing” a second server with requests? The server monitoring solution will blame the server that is being bombed by the server which has a runaway process running on it as opposed to indicating the source of the problem.

Real User Monitoring: An End to End Solution?

If you only take the user’s tier into account, real user monitoring could be considered an end-to-end solution for that tier alone. With real user monitoring, you can see both ends of a transaction—hence the claim to provide end-to-end. What about everything in the middle? What action is to be taken when a problem is detected by the end-user experience tool? What if the root of the problem resides within the database or the mainframe? Shouldn’t an end-to-end solution be able to take care of everything, including triage within the data center? Website performance monitoring is important for understanding the quality of service that your customers are receiving and there is a lot of value in that, but an end-to-end solution should also help you find the cause of the problem along with simply reporting its existence.

End to End Transaction Monitoring Tools Deliver

What do the following components have in common: the user’s desktop, a firewall, a proxy, a load balancer, a Web server, an application server, a message broker, a database and a mainframe? The answer is transactions. The only way to provide a true end-to-end solution is by monitoring every transaction from the moment any user clicks any button, and continuing all the way through each tier. Only business transaction management (BTM) solutions can promise that.

Performance monitoring tools that are not showing end-user performance are not focusing on what is most important to the business. Website monitoring tools that send synthetic transactions to the site and check response times are not showing what users are really experiencing. Database monitoring is important, but without knowing the context of that problematic SQL server transaction, resolution can be a shot in the dark.

Transaction Monitoring With BTM

End-to-end solutions must be able to monitor all of the infrastructure’s components. Transaction monitoring solutions do that automatically since they monitor the object that ties all of those different components together—the transactions. BTM solutions enable a drill down that begins from each transaction type that is running on the system and ends with the smallest event that composes a single transaction instance. In this manner, not only are you assured that everything is running smoothly, but when things start to go wrong you can perform immediate triage and resolution.

SharePath—The Most Comprehensive Transaction Monitoring Solution

SharePath by Correlsense is the only single solution on the market today that can provide true end-to-end transaction monitoring. From the moment a user clicks any button in a monitored desktop or Web-based application, SharePath monitors the transaction through the entire infrastructure, through proxies, Web servers, load balancers, application servers, message brokers, databases and mainframes. Learn more.

September 23, 2011

No Comments

Why is it always on a Friday? (Part 1)

The weekend is tomorrow; you’ve been head-down all week, nose to the grindstone making sure that your people and your apps are doing well , everything is running smoothly and your end-users are experiencing great service. Now is the time to go home, make sure your wife is OK and to “monitor” your family. Suddenly, on Friday afternoon you get the call – something is wrong… and it’s not one of the “easy ones” (they never are on Friday): you’re calling a war room. REALLY!?!?

It’s called the War Room for good reason – that’s just what it is. You’ve been attacked, and you need to fire back with everything you’ve got. The problem can be one of 100 different issues – latencies are cropping up in your transactions, end users are starting to get timed out, and are growing more and more frustrated. The pressure is on to solve the problem immediately. What’s driving you crazy is that all the screens are showing green! Nothing is indicating a problem anywhere along the pipe, so there’s no one to blame. DBA’s, Network, System Admin, app owners – they’re all happy with their dashboards and more than happy to show you. And that’s great for them, but with end-user transactions timing out, what’s to stop your customer from going straight to the competition?

In fact, end user experience monitoring would show that your users are just plain fed up. You and your teams need to figure out what, exactly, is going wrong.

IT can be a needy bunch. They work hard, and they’re stuck behind the scenes where they get very little accolade (thank God I at least have this blog to complain!). If things go well, end users never even think of them. In fact, the only time they get any attention at all is when things start to fall apart – and that’s just the way it should be.

So like children, your IT team is suddenly all over you. Each person is trying to prove that on their end, everything is fine. It’s green across the board. But you know that somewhere in between those green lights, there is a pile-up.

What are you going to do in this situation?

How do you save the operation from days of latency and lost customers?

July 14, 2011

No Comments

Google Tries to Keep Google Plus from Getting Overloaded

When Google released its new social networking tool, Google Plus last week, it was confined to what they called a limited invitation-only Beta. Scoring an invitation was a precious commodity, but many new members (including me) quickly found that instead of accessing the system, we found a sign that indicated the system was temporarily over capacity.

I’m guessing that Google wanted to make sure there wasn’t so much traffic, that it would take down the system. Not exactly what you want to do in the first days of release of a hot service. So instead, they throttled the number of new people they would allow on the system at any one time.

I’m speculating of course because I don’t have actual insight into what goes on in Google’s data centers, but I would be willing to bet they had sophisticated monitoring tools in place, maybe even tools they developed specifically for this type of case.

They knew they would have some potentially unhappy users who couldn’t get immediate gratification of getting into the social network, but they were probably willing to trade that temporary disappointment to ensure that the people who were already in would get the optimal Google Plus experience. If the system were slow, unavailable or balky, that would have been a bigger problem.

They were probably watching for the point at which the system performance began to decay, and if that wasn’t due to any number of factors they might understand and control, it could very well have to do with the number of people using the system. By shutting down new user access temporarily, it ensures, all things being equal, that the experience of current users wouldn’t be affected by a large influx of new users.

And it buys them time to increase capacity to accommodate the next wave of traffic. It’s a smart approach if that’s what they were doing, especially with a popular new service like Google Plus. Perhaps, Google also learned some lessons about how it rolled out the failed Google Wave, its first attempt at social networking, which they kept limited to a select few early adopters for so long, it never really caught on (of course, there were other factors that contributed to its failure too).

Whatever the reason, if Google is using monitoring tools in this fashion to keep the system operating optimally, it’s sensible way to ramp up capacity, while ensuring that the users on the system don’t feel any of the company’s growing pains as new users flock to the social network.

July 12, 2011

No Comments

Transaction Monitoring -The Four Approaches

This article discusses transaction monitoring and the 4 approaches of network appliances, Java/.NET deep dive, end-user measurements, and agent based monitoring.

You know that your enterprise is in dire need of a transaction monitoring solution. Transaction latencies are in the sky, some transactions do not even make it through, mysterious bottlenecks pop up during peak use, and users call in with complaints. Implementing a transaction monitoring solution is unavoidable.

The problem is, when you look at all of the transaction monitoring solutions out there they all look the same. Every vendor seems to make the same claims: end-to-end transaction monitoring, full visibility into transactions, link IT to business processes, low overhead, perform root cause analyses, find application performance problems before they impact your clients, and so on.

So, how can you make sense of all the claims and find the transaction monitoring solution that is right for you? This article reviews the various transaction monitoring approches and presents the advantages and disadvantages of each.

1. Transaction Monitoring: Four Solution Types

  • Network Appliances: Includes all solutions that collect data by non-intrusively listening to network traffic.
  • Deep Dive Profilers: J2EE/.NET code instrumentation. Includes all solutions that use bytecode instrumentation (or Java/.NET Hooks) in order to collect thorough code-level metrics.
  • End-User Monitoring: This approach utilized end-user latencies either by connecting an appliance to the network at the Web server or by installing agents on the client side.
  • Agent-based Transaction Tracing: Solutions where agents are installed on all relevant tiers and all transactions are traced everywhere.
  • Transaction Monitoring with Network Appliances
  • What they do: perform transaction monitoring by network sniffing in order to identify application events (or transaction segments), and then reconstruct the entire transaction by fuzzy matching.
  • How they do it: network appliance solutions usually connect to a port mirror in order to collect the traffic, and then try to reconstruct the entire transaction. Information needs to be collected directly from every node that is of interest.
  • Main advantage: transaction monitoring by connection to the network means that no overhead is added and it is simple to install.
  • Main drawback: transaction monitoring has to be done by an algorithmic approach since they cannot collect data directly from the servers, which leads to inaccurate transaction metrics and topology.

Correlsense SharePath vs. Network Appliance Solutions

SharePath can provide data from within the servers, enabling resource consumption and all of the actual parameters of the transaction segment to be collected without requiring fuzzy matching, which can be inaccurate. SharePath is much more flexible, is very non-invasive, and simply records what goes in and out of the server, and it can dive-in to collect metrics from encrypted data packets.

2. Transaction Monitoring with Deep Dive into the Java/.NET Code

  • What they do: these transaction monitoring tools provide deep diagnostics into Java applications to the code level. They are used by J2EE/.NET experts in order to locate problems before deployment.
  • How they do it: transaction monitoring is done by bytecode instrumentation (or Java hooks) that retrieve data from the nodes that are running J2EE/.NET applications. This is done by utilizing the class loading mechanism of the interpreter (JVM for J2EE or CLR for .NET) that, in order to intercept specific classes or methods, calls within the application.
  • Main advantage: provides a lot of rich information for developers. This type of transaction monitoring brings up the specific line of code that is the cause of the problem.
  • Main drawback: transaction monitoring cannot be done for all of the transactions running on the system (up to 10% for short periods of time), implementations are lengthy and invasive, and the person ultimately responsible for application performance may be completely overwhelmed and/or clueless by how to utilize the massive amounts of information retrieved.

SharePath vs. Deep Dive

Transaction monitoring by deep dive cannot load test at full capacity during development or monitor all transactions during production like SharePath can with its very low overhead. SharePath can be used with one of the many J2EE and .NET profilers that are available on the market today (some are for free) in order to aid application development before deployment. Because of its ease of use and high-level view, SharePath can be used by IT operations and infrastructure teams to trace transactions at all tiers, providing a full topology of the transaction flow. Conversely, deep dive solutions provide metrics only at the application server, and they have limited horizontal visibility. Lastly, SharePath’s architecture is environment-independent and can be deployed on any server, not just the .NET or J2EE server. The product’s technology is based on process wrapping and, therefore, is non-invasive, which enables fast and clean implementation.

3. Transaction Monitoring with the Help of End-User Measurements

  • What they do: this transaction monitoring approach is based on managing the application by monitoring the end-user response times and then performing customer analytics and system heuristics from the Web server outward.
  • How they do it: there are two general strategies to implement transaction monitoring with this approach. The first is by installing an agent on the user’s computer, either on the desktop or in the browser with the help of a Java script. The second is by installing a network appliance on the Web server. Some solutions have servers around the world that test performance from different regions.
  • Main advantage: provides valuable metrics about what your customers are experiencing. Transaction monitoring with this approach puts customers first.
  • Main drawback: transaction monitoring stops at the Web server. The few solutions that let you peek beyond that can only provide very limited metrics. You will know that there is a problem, but you will have no idea where to find it.

SharePath vs. End-User Monitoring

SharePath can be extended with an end-user monitoring solution in order to cover all bases. The product finds the specific location of problems versus only alerting about them, providing important insight and saving time. Real end-user monitoring products do not provide much information in the development phase. However, with SharePath, you can test your application pre-deployment.

4. Agent-based Transaction Monitoring

  • What they do: software agents are deployed along the application path across the different tiers so that a unified view of the entire application is provided. What characterizes these solutions is that the full flow of every single one of the transactions running on the system is recorded in real time, at thousands of transactions a second. This solution is just as valuable to IT management as it is for the application development team.
  • How they do it: agents installed at each tier send data collected to a central repository for processing. Agents may be installed with the help of JVM/CLR instrumentations at the application server (one technical execution approach), or they may be installed as kernel modules (drivers), shared objects (Unix), or DLL files (Windows) at the operating system level. Agents may also be installed at databases, MQ middleware servers and legacy mainframe computers. Every event or transaction segment is recorded along with all of its real parameters and then accurately re-constructed.
  • Main advantage: transaction monitoring is done all the time for every single transaction. This is the only true end-to-end solution that includes the middle and all of the important data that is to be collected from the servers in a way that does not weigh down the system.
  • Main disadvantage: by themselves, these transaction monitoring solutions cannot get deep into the code or see what is happening at the browser level.

SharePath vs. Other BTM Solutions

SharePath’s deployment is faster and is not as labor intensive as other BTM solutions that need to perform code instrumentations—for example, SharePath installs drivers, DLL files, objects, etc. SharePath can be easily used by anyone, and it works at the operating system level (wraps processes as opposed to bytecode instrumentation) and is therefore environment independent. Residing at the OS level means that SharePath can better monitor resource consumption beyond CPU consumption (IO, network, and can be customized to collect anything). View videos.

June 9, 2011

No Comments

Transaction Monitoring: Traditional vs. BTM

Transaction monitoring includes both traditional and new approaches. Before making a purchase, read this article to learn the differences and make the right choice.

Even experienced IT professionals can confuse traditional monitoring tools and the newer generation of business transaction monitoring (BTM) tools. In today’s market, competing messages sound very similar, making it difficult to differentiate between the old and the new. This article highlights the differences in the traditional and the newer generation of transaction monitoring tools.

Traditional Monitoring

Traditional tools monitor the performance of each component individually and display all of these metrics on a “single pane of glass.” End-to-end performance monitoring means that you can see the performance of every component in one centralized console. So, for example, the resource consumption of the servers, the threads that the application is running, the throughput of the network components, and the calls to the database all displayed in their own section.

When traditional tools monitor transactions, they pick up various segments of transactions throughout the data center without stitching them together into one full transaction flow.

For example, the database monitor picks up all of the SQL statements that it sees and displays them on the central dashboard along with their response times, while the real user monitor picks up all of the requests that are sent out to the data center and displays them on the same dashboard along with their response times. If an application slowdown occurs and all monitors (including the application server monitor which was not mentioned) are showing erratic response times for various transactions, the real user measurements only show that the user is experiencing a problem but does not show where the problem is within the data center. The silo-specific tools, then, do not have the context of the CICS program names, SQL statements, and the Web service calls which are showing erratic performance. The result is that the IT professional is stuck with a glut of confusing, disparate information on the dashboard.

How to Identify Traditional Monitoring Tools

  • They are typically sold as product suites. Vendors develop or acquire separate server monitoring tools, network monitoring tools, application performance management tools and real user measurement tools, and then offer them bundled as an end-to-end package.
  • These tools tend to be pricey and are difficult to implement, not to mention their limited visibility due to the lack of correlation between tiers. On the upside, they can provide more thorough metrics within the application server, which is why the new generation seeks to complement the traditional tools as opposed to replacing them.

The New Generation of BTM Tools

The new BTM tools connect every process within the data center to a click of a user at the desktop.

End-to-end means that the user request and the related activity within the proxy, Web server, app server, database server, MQ and mainframe are all connected as a single transaction instance. The resource consumption at each component can still be seen, but at the granularity of a single transaction segment.

For example, if service levels are starting to degrade, the new generation of tools not only picks up the performance degradation that the user is experiencing, but they also immediately know what is causing the specific degradation down the line.

Learn more about how SharePath—the new generation of monitoring—can help your enterprise thrive in managing today’s complex applications.

December 30, 2010

No Comments

End-to-End Transaction Monitoring

End-to-end transaction monitoring solutions should monitor all of the infrastructure’s components, yet some solutions only monitor one part of the bigger picture.

The term “end-to-end” relative to transaction monitoring is very over-used. End-to-end Website monitoring, end-to-end server performance monitoring, end-to-end database monitoring, and end-to-end network performance monitoring. The list can go on and on when, in fact, they only provide monitoring for one small part of the bigger picture, as illustrated below.

Why True End-to-End Transaction Monitoring is Different than Monitoring Server Performance

Monitoring server performance traditionally refers to ensuring that CPU utilization and memory consumption have not reached maximum levels. However, this does not promise true application availability. Traditional end-to-end monitoring tools provide a dashboard that shows the health of each one of the individual components within the data center. But what if someone forgot to re-connect a network cable after maintenance or what if a load balancer was configured incorrectly and is not executing a proper round robin? An alert will indicate that something is wrong with the servers that are still connected when, in fact, the problem is that traffic is simply not being equally distributed. A network performance monitoring solution could be implemented, which would aid by monitoring the usage of the network, but that is not a single end-to-end solution anymore.

What if a runaway process in one server is “bombing” a second server with requests? The server monitoring solution will blame the server that is being bombed by the server which has a runaway process running on it as opposed to indicating the source of the problem.

Real User Monitoring: An End to End Solution?

If you only take the user’s tier into account, real user monitoring could be considered an end-to-end solution for that tier alone. With real user monitoring, you can see both ends of a transaction—hence the claim to provide end-to-end. What about everything in the middle? What action is to be taken when a problem is detected by the end-user experience tool? What if the root of the problem resides within the database or the mainframe? Shouldn’t an end-to-end solution be able to take care of everything, including triage within the data center? Website performance monitoring is important for understanding the quality of service that your customers are receiving and there is a lot of value in that, but an end-to-end solution should also help you find the cause of the problem along with simply reporting its existence.

End to End Transaction Monitoring Tools Deliver

What do the following components have in common: the user’s desktop, a firewall, a proxy, a load balancer, a Web server, an application server, a message broker, a database and a mainframe? The answer is transactions. The only way to provide a true end-to-end solution is by monitoring every transaction from the moment any user clicks any button, and continuing all the way through each tier. Only business transaction management (BTM) solutions can promise that.

Performance monitoring tools that are not showing end-user performance are not focusing on what is most important to the business. Website monitoring tools that send synthetic transactions to the site and check response times are not showing what users are really experiencing. Database monitoring is important, but without knowing the context of that problematic SQL server transaction, resolution can be a shot in the dark.

Transaction Monitoring With BTM

End-to-end solutions must be able to monitor all of the infrastructure’s components. Transaction monitoring solutions do that automatically since they monitor the object that ties all of those different components together—the transactions. BTM solutions enable a drill down that begins from each transaction type that is running on the system and ends with the smallest event that composes a single transaction instance. In this manner, not only are you assured that everything is running smoothly, but when things start to go wrong you can perform immediate triage and resolution.

SharePath—The Most Comprehensive Transaction Monitoring Solution

SharePath by Correlsense is the only single solution on the market today that can provide true end-to-end transaction monitoring. From the moment a user clicks any button in a monitored desktop or Web-based application, SharePath monitors the transaction through the entire infrastructure, through proxies, Web servers, load balancers, application servers, message brokers, databases and mainframes. Learn more.