Anyhow, so we were a bunch of technology companies presenting in this huge ball room. Each company got seven minutes to present what it has to offer. Now, you can say many things about me, but if there is something I really understand it is IT technology. And still, there were at least ten companies which I couldn’t understand what they were selling. Really. I listened carefully to ALL the presentations (not an easy task), and still there were these sales rep’s presenting their offering, and sometimes even the CEO or founder of the company, and they simply couldn’t describe what they were selling in a manner that I’ll be able to understand. UNBELIEVABLE. When it comes to Sales guys I’m not surprised, most of them really don’t know what their product actually does…The best sales guys I’ve met were selling shoes and encyclopedias before moving to high-tech…

It reminded me of a story a veteran salesman once told me:

He had this huge account, and the morning of the presentation, he heard that his competition had a long presentation a day before, and that they had left the customer with a bunch of books about the product. So when the presentation started he said : “There are those that write books about their product. You should ask yourself why. We simply have a working, useful, product. Please allow me to show it to you…” He won the deal eventually.

A week before the actual conference, we decided that we would use our seven minutes to show a demo of our product. Simply show what it does. Isn’t that the idea?? I guess not everyone in my industry still gets it – I believe we should simply create products that:

1. Work
2. Do what they claim to do
3. Do something the customer needs

All the rest is just throwing salt in your customers’ eyes.

…and my guess is that they are itching already.

The first thing we wanted to make sure was that the new wiring to our communication racks was working. We took the wireline technician to the communication room where there was this cable hanging from the ceiling with a phone end-point, so we automatically assumed that it is one of the phone lines in the office. Next step was to find the hub that is wired to this phone line, so the technician can define the phone line as ours and enable it for internet. So we\’re looking around in our offices which was then filled with construction workers and paint brushes and we can\’t find the hub. The technician remembered that he already did some wiring work in the building, and he thinks it\’s in the other side of the building in one of the offices. We kept on looking in different places, with no success, and eventually decided we need to enter one of the abandoned office. But the building maintenance guy left already. No keys. End of first session.\r\n

The next day I verified the key to the office, and we immediately found the hub and the technician got to work. We then returned to our space to see if the phone line is enabled. It wasn\’t. At this point I waited for him, and he kept on hopping between the hub we found (which was in the other side of the building…a five minutes walk each time…and it was hot J until he told me that it just doesn\’t work and we\’ll to buy new infrastructure for the phone lines. So I shouted a bit, and talked to his manager, and eventually someone in the phone company told them there might be another hub in another office, the one next door to us. We thrilled with joy, but of course it was already late. Again, no keys. End off second session.\r\n

The next day, we went to our neighbors, and indeed found another hub. The technician was exhilarated, and started working again. The same evening I got a call that there is still no active phone line. Again – we\’ll have to lay down new phone line wires – but not from the other side of the building, just from the neighbors hub. End of third session.\r\n

At this point, I was starting to feel that we were missing something. I went late at night to our new space, after all the handymen left, and entered the communications room. I took the phone line that we assumed IS the wired phone line, and started to track its trail. So I started to pull it, and I pulled and pulled – the phone line turned out in my hand. I mean the other side of the phone line turned out in my hand. IT WASN\’T CONNECTED TO ANYTHING. NO WONDER THE TECHNICIAN COULDN\’T GET THE PHONE LINE TO WORK!\r\n

I kept looking in the ceiling, and eventually found a different cable that looked like a phone line, and seemed as if was coming from our next door neighbors. Of course this one turned out to be the REAL phone line.\r\n

It immediately reminded me of the first thing I do when I get called to a customer site during a malfunction of some kind – I ask \”What\’s your current goose chase?\” – which means: what is the current theory you are trying to validate, although there is no special reason to pursue it, and the only reason you ARE pursuing it is that you have to do something, otherwise your managers/colleagues will start to think that you don\’t have a clue, which is exactly the situation…\r\n

It usually happens with malfunctions in multi-tier applications. These events involve so many different technical experts, that goose chases are bound to happen. For example, I once got called to a customer, whose application was freezing 10 times a day. It happened in his online critical secured financial application – which uses SSL of course – and it didn\’t happen in his online content website, which was not SSL encrypted. Each application had a different app server, so there were a zillion possible reasons for the problem, but nevertheless the system guys came with their own theory: There are problems where SSL is used, and there aren\’t problems were SSL in not used. So it is the SSL! Why not upgrade the SSL accelerator? Yeah! That\’s a good idea! Let\’s do that!\r\n

And believe it or not – that\’s what they did, and when I came it was after a couple of work days already invested in upgrading this SSL accelerator, that OF COURSE had NOTHING to do with the actual root-cause.\r\n

My advice – define the phenomena as accurately as you can, and then try to collect as much useful data as you can. Transaction response times broken down by tiers, resource usage within each tier, all available logs from each tier, throughput of each tier, and it all has to be collected in the granularity of the problem – the average behavior in the last 5 minutes probably won\’t cut it…Once all the data is available, and aligned according to a time line, there are many questions that can be answered. It\’s always astonishing to me that this basic first step is usually not implemented. If you don\’t have the data – invest time in creating/deploying some useful probes that will collect the data you need, and don\’t forget to do so during the malfunction!\r\n

Beware of Goose Chases! \r\n

Cheers, Lanir.

So we are also in the middle of a round, as always, and have brought aboard an ‘old school’ investor- Very knowledgeable regarding our business and how to build a healthy and stable company. It’s interesting to see the difference between different types of investors – some use excel and drill down to fine precisions, while others just use ‘Gut Feeling’. This is exactly what 2002 noble prizewinners, Daniel Kahneman and Vernon Smith, studied for a long time –behavior based on prior experience of some kind. They discovered that human decisions, rather than being based on a full analysis of the situation, often rely on shortcuts or rules of thumb. So for us, the measure freaks that measure and calculate everything that moves – it’s often hard to deal with ‘Gut Feeling’.

However, it reminded me that I see the same thing when there are performance problems. Usually, there are no well defined measures of what is the desired performance and what we should compare to. And I’m referring to sites that do have SLA measurement of some kind, but almost all of them don’t have a RUM (Real User Measurement) solution, so they rely on synthetic robots of some kind to measure their performance. But these synthetic measurements don’t always reflect what is going on in the entire data center. And you’ll be surprised how much organizations still don’t have constant SLA measurement of their entire services, synthetic or not.

So how can you solve a performance problem when you can’t know for sure what was before and what you should aim to? Usually, our first step is to implement a quick service level measurement solution – either using our SLAce / SharePath products, or (if the customer is cheap) a simple synthetic measurement - and wait for the problem to re-occur. Then we try to set expectations with the customer, regarding the level of improvement we can deliver.

But how can we know how much we can improve if we don’t have a clue regarding how it was before the problem??

Experience, expert knowledge, complex analysis of all the data we collected until the phenomena occurred again – or in other words –’Gut Feeling’…

Cheers, Lanir.

Ten days later (I kid you not!) the guy from the washing machine company came to fix the problem. I told him there was a leak from the tap. He started disassembling the washing machine and explained to me in detail how I should always keep this compartment clean and that compartment clean (where you put the washing liquid) and I’m like: yeah, yeah, just solve the goddamn leak so we can wash our clothes, man. By this time the family started to wear very unfashionable clothes…and new dirty cloths got dumped in a pile. You can say I suffered from this malfunction. He worked on it for like 30 minutes, cleaned this and cleaned that, told me that that’s the reason I had a leak from the tap, turned the machine on and everything seemed to be OK. No tap leaked. However, once the machine started to drain the clothes, meaning water got poured out to the pluming hole - water started to fill the room. The plumbing was blocked.

The way the water filled the room never happened before this maintenance guy came and started playing with it. However, it really did not seem to have anything to do with him. And it was definitely a problem. So maybe this is the reason there was a pond behind my washing machine in the first place, and not the leak from the tap. I did not know the answer, but I worked on solving the pluming block myself. Eventually the washing machine returned to work, no water was pouring out to the kitchen floor, and there was no immediate leak from the tap as before. Problem solved.

It is amazing how IT applications act the same as washing machines. I don’t know why, it’s beyond me, but in the really hard performance problems we had to deal with in our DoctorIT service, there was a clear phenomena which the customer suffered from (poor response times, a performance bottleneck etc.), and there was a root-cause we found that beautifully explained the phenomena. The customer swore that there was no other performance problem before, and since we isolated a well-defined root cause – we were certain that we found and solved the performance problem. However, that did not happen.

Can you guess why?

Because the really hard-to-detect problems - the ones that are tricky and keep giving you a hard time to detect and isolate them –come in PAIRS. They don’t like to play alone. It’s not fun for them. They need a friend. The moment you put your hands on one of them, the other hits you from behind. You tend to forget they are a pair, and all your calculations and assumptions focus on isolating a single one, but that is the worst mistake you can make.

First lesson in solving hard Heisenbugs: assume, from the beginning, that there is more than one performance bottleneck.

If you are lucky, there is only a single performance bottleneck, and your work will be fast and smooth. However, you have to assume there is more than one of these application performance problems, and then you won’t be surprised. This way you can better manage your resources for solving the actual bottlenecks and your customer’s expectations will be set accordingly.

So let’s get a little bit technical and describe a real case study of detecting and isolating a performance bottleneck in an enterprise application:

We were called in after the customer has been suffering from poor application performance and response times for over a month. The application would freeze or hang for a few minutes, and then return to work smoothly. The application hangs happened once a week on average, with no obvious pattern, and no code change that could hint where the performance problem might be. We recorded the transaction traffic and resource usage of the application (millions of HTTP requests and SQL statements per day, for a period of week), until the application hanged again. Resource consumption analysis revealed that the poor application performance was originally due to memory consumption in the application server…

which then caused other resources to be highly utilized…

Lack of memory caused a high paging rate…

which caused a high read/write disk
queue…

which was the immediate cause for the performance
bottleneck and poor response times.

Our theory was there was a single instance of an http request that triggered an event that over-consumed memory in the application server. We also assumed this event started minutes before the actual response time degradation occurred. We searched for a long running transaction that started 5-10 minutes before the poor response times began, and lasted at least until 10-20 seconds before the application hanged. We indeed found one! We rushed back to the customer site and managed to recreate the transaction, and cause the performance problem all over again. Using specific parameters, this transaction would cause enormous memory consumption. The problem was quickly detected, isolated, and fixed. The customer was happy, we were happy, and our bank account was almost happy.

Two weeks later, the customer called again, and notified us that the problem happened again. How can that be? We knew for sure it was fixed! Well, to make a long story short, turned out there was another transaction, which under specific parameters generated hundreds of SQL queries that loaded the database, which also began in the same month.

Different root cause entirely - same phenomena. We should have known - Double Trouble.

By the way, my washing machine still leaks from time to time. God knows why.

Until next time, Lanir.

I guess I should introduce myself: My Name is Lanir Shacham, 32, father, husband, friend, surfer and CTO and founder of Correlsense. I started out with computers when I was seven playing on an old Sinclair, then Commodore (and Atari), Apple 2E, IBM PC (I was 13). I wrote my first program using BASIC, and I still remember how proud I was when I embedded a machine code procedure for drawing bricks that my brother wrote for me in my first self developed “BRICKS” game for Apple 2E. I was 10 years old. When I was 13 I stopped for doing mostly sports, sex, drinking and smoking until the age of 18 when I was recruited to the Israeli Army Computer Science Academy and started out my career. This is where I worked on mainframe machines writing PL1 programs using DB2 executed with JCL jobs, later on I worked on Digital (remember them?) Vax VMS servers (The best ever) with Oracle 6/7 using Forms 3.0 and Fortran/Pascal using SNA gateways to interact with a mainframe over WAN, then writing C/VB/Dev2K/PLSQL code for Win95, moving on to C, C++ and MS DNA framework (1997), then using Java and J2EE using Weblogic 2.0 (no WebSphere yet) and all of that before I was 23. I initiated and managed Israel intelligence corps’ information system as a young officer, designed together with my partner and friend Oren Elias, which is Correlsense’s CEO, and with some more colleagues and friends from the Performance Management industry, guys from Precise (today Symantec) , Quest Software, SAP, Oracle. Many top executives and founders of gorilla companies such as SAP, EMC, BMC, HP Mercury ,Checkpoint, Amdocs, Comverse and more came from the same school I grew up in. Many of Israel’s most successful software companies seeded out of ideas thought by bored programmers in the army. I spent six very long years in Israel’s intelligence force IT department but I can surely state that it was, and still is, by far the best school in our industry. Since then I did a lot more stuff, probably working on every existing platform to date – I dare you to surprise me – I can compile code in any programming language, and if I’m not familiar with it then give me an internet connection and a couple of hours and you’ll get what you asked for. My last position was an IT Architect for IBM Global Technology Unit, consulting about architecture and middleware for the entire Israeli based software vendors. We have here the second biggest population of Hi-Tech companies after the valley, so I got to meet almost everyone, and I learned even more. I hold a B.Sc in Computer Science and M.Sc in Physics from Bar-Ilan university, both I accomplished while working full time. Kids – don’t try this at home – if you want to go to college or university, best to work only part time during that period or not to work at all!

I was a young officer with a budget of millions when I was 21, and I bought applications/middleware/hardware from all the vendors, later on I developed applications and was part of young companies trying to sell software, then I was the gorilla that everybody comes to listen to, but they don’t know that his products hang and crash (except MQ Series which is a fine product, right Sam?), or maybe they do know but just don’t care enough because nobody gets fired for buying Blue. And Don’t think that Red from Oracle is any different – it’s NOT.

WOW – that was a lot babbling about myself, but no more, it was just so you will show some respect and appreciate what I have to say, because when I speak badly about our industry – it is only because I AM the industry. I know it inside out.

I just read this morning an article about this guy David Platt, who is one of Microsoft’s “software legends” or something like that. Anyways, this guy is giving presentations these days in conventions and the topic is – drums – “software industry sucks!”. Programmers are lazy, unprofessional and produce terrible code! Well that was refreshing! I couldn’t agree more, and this is the point I want to make today – Why the hell are we as users are willing to work with no good products? Why? Well you can tell me that Windows was embraced by the mass, by I’m not talking about the mass. I’m talking about the products WE as an industry are using – our databases, app severs, system management tools and so forth – almost all of them are hard to install, have terrible UI, terrible logs and never behave as written in the manual. They disrespect us – the professionals who make this world tick. Applications tend to crash, hang, suffer from slow response time and poor performance, CPU jumps to 100%, memory gets fully used, disks fully utilized and what do we do? Carry ourselves with a dumb look on our faces as we wait for the system to reboot and hope for the best. Well I got really sick of it during my years in the big blue, and don’t think for a moment that IBM is any different from all of the other top vendors – CA, HP, BMC, Microsoft, Oracle – they all sell very bad products, and everybody are buying them. Why why tell me why??

Because they are GORILLAS selling to other GORILLAS and closing deals in the golf course instead of the lab. Well I am a goddam freakin’ monkey, and proud of it! And I want us monkeys to overthrow the slow, lazy, banana-eating gorillas who are inflating our daily jobs with mediocre experiences.

This is exactly the reason I founded Correlsense three years ago. I wanted to make a difference, or at least to try. And of course make a buck if I can while doing it Making a difference is by creating high quality software – the kind that you know is high quality from the moment you touch it. Making a difference is about solving problems to IT staff, it’s about educating your colleagues, your partners, your customers, and now you – on how you can become a better man in your job – by knowing what is bad practice and what is good, how to think when developing and maintaining an application, how you can solve hard performance problems instead of banging your head in the screen in front of you – I will tell you stories that will make you laugh and cry; I will make fun of those that deserve it and praise those that are worthy; I will expose the truth about code and architecture, about fuzzy algorithms that never work, about investors and top execs that try to look smart but just turn out dumb in the process, about arrogant programmers that don’t have a clue and much much more coming soon to a theater near you

You are right - I am terribly obnoxious - but I promise to be honest and at least tell the truth as I see it. No more, no less.

See you next time (if you dare), Lanir