The 21st Century Plumber’s Blog

Never been to the NYSE. There were times I was visiting Manhattan once a month and Wall Street in general, but still – never been to the New York Stock Exchange. So of course I was thrilled that the annual “Banking on IT” was held in the NYSE. Selected companies presented to IT executives from the banking industry, and we were lucky to be one of them. The organizers of the convention, the Chambers of commerce, told us that there is a dress code – suite and tie. Well I don’t have a tie. Were I come from – you wear a tie to funerals. Na, just kidding. You simply don’t wear a tie. But I was happy for the chance to dress up, once in my life. Although I was constantly worried I’ll spill some soda or cake on my fresh good looking suit and tie. Imagine how annoyed I was when one of the participants simply went in wearing a simple polo shirt. So the polo shirt had the company logo – Big deal! If I knew that was allowed, I’d staple a business card to my jeans …

Anyhow, so we were a bunch of technology companies presenting in this huge ball room. Each company got seven minutes to present what it has to offer. Now, you can say many things about me, but if there is something I really understand it is IT technology. And still, there were at least ten companies which I couldn’t understand what they were selling. Really. I listened carefully to ALL the presentations (not an easy task), and still there were these sales rep’s presenting their offering, and sometimes even the CEO or founder of the company, and they simply couldn’t describe what they were selling in a manner that I’ll be able to understand. UNBELIEVABLE. When it comes to Sales guys I’m not surprised, most of them really don’t know what their product actually does…The best sales guys I’ve met were selling shoes and encyclopedias before moving to high-tech…

It reminded me of a story a veteran salesman once told me:

He had this huge account, and the morning of the presentation, he heard that his competition had a long presentation a day before, and that they had left the customer with a bunch of books about the product. So when the presentation started he said : “There are those that write books about their product. You should ask yourself why. We simply have a working, useful, product. Please allow me to show it to you…” He won the deal eventually.

A week before the actual conference, we decided that we would use our seven minutes to show a demo of our product. Simply show what it does. Isn’t that the idea?? I guess not everyone in my industry still gets it – I believe we should simply create products that:

1. Work
2. Do what they claim to do
3. Do something the customer needs

All the rest is just throwing salt in your customers’ eyes.

…and my guess is that they are itching already.

There was no choice – we had to move. Although there is nothing more annoying than moving to new offices, we are growing J It took a while, but finally we found our new space and we had to do a little refurnishing before the actual move. The previous tenants of our new space, for some reason, decided they should take with them their entire data center infrastructure - they literally cut all the internal communication cables, in a brutal and strange manner, so they can take their racks with them…it is as if someone told the movers to \’take the racks\’!\r\n

The first thing we wanted to make sure was that the new wiring to our communication racks was working. We took the wireline technician to the communication room where there was this cable hanging from the ceiling with a phone end-point, so we automatically assumed that it is one of the phone lines in the office. Next step was to find the hub that is wired to this phone line, so the technician can define the phone line as ours and enable it for internet. So we\’re looking around in our offices which was then filled with construction workers and paint brushes and we can\’t find the hub. The technician remembered that he already did some wiring work in the building, and he thinks it\’s in the other side of the building in one of the offices. We kept on looking in different places, with no success, and eventually decided we need to enter one of the abandoned office. But the building maintenance guy left already. No keys. End of first session.\r\n

The next day I verified the key to the office, and we immediately found the hub and the technician got to work. We then returned to our space to see if the phone line is enabled. It wasn\’t. At this point I waited for him, and he kept on hopping between the hub we found (which was in the other side of the building…a five minutes walk each time…and it was hot J until he told me that it just doesn\’t work and we\’ll to buy new infrastructure for the phone lines. So I shouted a bit, and talked to his manager, and eventually someone in the phone company told them there might be another hub in another office, the one next door to us. We thrilled with joy, but of course it was already late. Again, no keys. End off second session.\r\n

The next day, we went to our neighbors, and indeed found another hub. The technician was exhilarated, and started working again. The same evening I got a call that there is still no active phone line. Again – we\’ll have to lay down new phone line wires – but not from the other side of the building, just from the neighbors hub. End of third session.\r\n

At this point, I was starting to feel that we were missing something. I went late at night to our new space, after all the handymen left, and entered the communications room. I took the phone line that we assumed IS the wired phone line, and started to track its trail. So I started to pull it, and I pulled and pulled – the phone line turned out in my hand. I mean the other side of the phone line turned out in my hand. IT WASN\’T CONNECTED TO ANYTHING. NO WONDER THE TECHNICIAN COULDN\’T GET THE PHONE LINE TO WORK!\r\n

I kept looking in the ceiling, and eventually found a different cable that looked like a phone line, and seemed as if was coming from our next door neighbors. Of course this one turned out to be the REAL phone line.\r\n

It immediately reminded me of the first thing I do when I get called to a customer site during a malfunction of some kind – I ask \”What\’s your current goose chase?\” – which means: what is the current theory you are trying to validate, although there is no special reason to pursue it, and the only reason you ARE pursuing it is that you have to do something, otherwise your managers/colleagues will start to think that you don\’t have a clue, which is exactly the situation…\r\n

It usually happens with malfunctions in multi-tier applications. These events involve so many different technical experts, that goose chases are bound to happen. For example, I once got called to a customer, whose application was freezing 10 times a day. It happened in his online critical secured financial application – which uses SSL of course – and it didn\’t happen in his online content website, which was not SSL encrypted. Each application had a different app server, so there were a zillion possible reasons for the problem, but nevertheless the system guys came with their own theory: There are problems where SSL is used, and there aren\’t problems were SSL in not used. So it is the SSL! Why not upgrade the SSL accelerator? Yeah! That\’s a good idea! Let\’s do that!\r\n

And believe it or not – that\’s what they did, and when I came it was after a couple of work days already invested in upgrading this SSL accelerator, that OF COURSE had NOTHING to do with the actual root-cause.\r\n

My advice – define the phenomena as accurately as you can, and then try to collect as much useful data as you can. Transaction response times broken down by tiers, resource usage within each tier, all available logs from each tier, throughput of each tier, and it all has to be collected in the granularity of the problem – the average behavior in the last 5 minutes probably won\’t cut it…Once all the data is available, and aligned according to a time line, there are many questions that can be answered. It\’s always astonishing to me that this basic first step is usually not implemented. If you don\’t have the data – invest time in creating/deploying some useful probes that will collect the data you need, and don\’t forget to do so during the malfunction!\r\n

Beware of Goose Chases! \r\n

Cheers, Lanir.

A startup company always needs more money. This is the basic truth of our existence. No matter how much money you raised – you need more – you need to expand, bring more business, more employees to support the new business – and that costs money, new money that usually you still don’t have

So we are also in the middle of a round, as always, and have brought aboard an ‘old school’ investor- Very knowledgeable regarding our business and how to build a healthy and stable company. It’s interesting to see the difference between different types of investors – some use excel and drill down to fine precisions, while others just use ‘Gut Feeling’. This is exactly what 2002 noble prizewinners, Daniel Kahneman and Vernon Smith, studied for a long time –behavior based on prior experience of some kind. They discovered that human decisions, rather than being based on a full analysis of the situation, often rely on shortcuts or rules of thumb. So for us, the measure freaks that measure and calculate everything that moves – it’s often hard to deal with ‘Gut Feeling’.

However, it reminded me that I see the same thing when there are performance problems. Usually, there are no well defined measures of what is the desired performance and what we should compare to. And I’m referring to sites that do have SLA measurement of some kind, but almost all of them don’t have a RUM (Real User Measurement) solution, so they rely on synthetic robots of some kind to measure their performance. But these synthetic measurements don’t always reflect what is going on in the entire data center. And you’ll be surprised how much organizations still don’t have constant SLA measurement of their entire services, synthetic or not.

So how can you solve a performance problem when you can’t know for sure what was before and what you should aim to? Usually, our first step is to implement a quick service level measurement solution – either using our SLAce / SharePath products, or (if the customer is cheap) a simple synthetic measurement - and wait for the problem to re-occur. Then we try to set expectations with the customer, regarding the level of improvement we can deliver.

But how can we know how much we can improve if we don’t have a clue regarding how it was before the problem??

Experience, expert knowledge, complex analysis of all the data we collected until the phenomena occurred again – or in other words –’Gut Feeling’…

Cheers, Lanir.

So everybody’s writing a blog these days. “Write a CTO blog” Oren told me, “Everybody does”. Why? Do you really need to share your thoughts with everyone out there? Says who? I’m afraid someone out of my inner conscious will know my thoughts…especially not my colleagues /employees/partners/investors/competitors in the industry. Man, that won’t be such a good thing for me. ‘Cause honestly – I really don’t think highly of our industry these days. Like Sturgeon’s law states: “90% of everything is crap”. Well, probably not everything – I mean Google makes my life a lot easier although I’m starting to get really terrified from the control they have over information, and VMWare/Xen – these guys did a hell of a job! I have 50 linux servers working on five simple Intel machines that cost me all together 5K$… This is a revolution in front of our eyes. We are returning to the mainframe days…Kids, do you know what a mainframe is?

I guess I should introduce myself: My Name is Lanir Shacham, 32, father, husband, friend, surfer and CTO and founder of Correlsense. I started out with computers when I was seven playing on an old Sinclair, then Commodore (and Atari), Apple 2E, IBM PC (I was 13). I wrote my first program using BASIC, and I still remember how proud I was when I embedded a machine code procedure for drawing bricks that my brother wrote for me in my first self developed “BRICKS” game for Apple 2E. I was 10 years old. When I was 13 I stopped for doing mostly sports, sex, drinking and smoking until the age of 18 when I was recruited to the Israeli Army Computer Science Academy and started out my career. This is where I worked on mainframe machines writing PL1 programs using DB2 executed with JCL jobs, later on I worked on Digital (remember them?) Vax VMS servers (The best ever) with Oracle 6/7 using Forms 3.0 and Fortran/Pascal using SNA gateways to interact with a mainframe over WAN, then writing C/VB/Dev2K/PLSQL code for Win95, moving on to C, C++ and MS DNA framework (1997), then using Java and J2EE using Weblogic 2.0 (no WebSphere yet) and all of that before I was 23. I initiated and managed Israel intelligence corps’ information system as a young officer, designed together with my partner and friend Oren Elias, which is Correlsense’s CEO, and with some more colleagues and friends from the Performance Management industry, guys from Precise (today Symantec) , Quest Software, SAP, Oracle. Many top executives and founders of gorilla companies such as SAP, EMC, BMC, HP Mercury ,Checkpoint, Amdocs, Comverse and more came from the same school I grew up in. Many of Israel’s most successful software companies seeded out of ideas thought by bored programmers in the army. I spent six very long years in Israel’s intelligence force IT department but I can surely state that it was, and still is, by far the best school in our industry. Since then I did a lot more stuff, probably working on every existing platform to date – I dare you to surprise me – I can compile code in any programming language, and if I’m not familiar with it then give me an internet connection and a couple of hours and you’ll get what you asked for. My last position was an IT Architect for IBM Global Technology Unit, consulting about architecture and middleware for the entire Israeli based software vendors. We have here the second biggest population of Hi-Tech companies after the valley, so I got to meet almost everyone, and I learned even more. I hold a B.Sc in Computer Science and M.Sc in Physics from Bar-Ilan university, both I accomplished while working full time. Kids – don’t try this at home – if you want to go to college or university, best to work only part time during that period or not to work at all!

I was a young officer with a budget of millions when I was 21, and I bought applications/middleware/hardware from all the vendors, later on I developed applications and was part of young companies trying to sell software, then I was the gorilla that everybody comes to listen to, but they don’t know that his products hang and crash (except MQ Series which is a fine product, right Sam?), or maybe they do know but just don’t care enough because nobody gets fired for buying Blue. And Don’t think that Red from Oracle is any different – it’s NOT.

WOW – that was a lot babbling about myself, but no more, it was just so you will show some respect and appreciate what I have to say, because when I speak badly about our industry – it is only because I AM the industry. I know it inside out.

I just read this morning an article about this guy David Platt, who is one of Microsoft’s “software legends” or something like that. Anyways, this guy is giving presentations these days in conventions and the topic is – drums – “software industry sucks!”. Programmers are lazy, unprofessional and produce terrible code! Well that was refreshing! I couldn’t agree more, and this is the point I want to make today – Why the hell are we as users are willing to work with no good products? Why? Well you can tell me that Windows was embraced by the mass, by I’m not talking about the mass. I’m talking about the products WE as an industry are using – our databases, app severs, system management tools and so forth – almost all of them are hard to install, have terrible UI, terrible logs and never behave as written in the manual. They disrespect us – the professionals who make this world tick. Applications tend to crash, hang, suffer from slow response time and poor performance, CPU jumps to 100%, memory gets fully used, disks fully utilized and what do we do? Carry ourselves with a dumb look on our faces as we wait for the system to reboot and hope for the best. Well I got really sick of it during my years in the big blue, and don’t think for a moment that IBM is any different from all of the other top vendors – CA, HP, BMC, Microsoft, Oracle – they all sell very bad products, and everybody are buying them. Why why tell me why??

Because they are GORILLAS selling to other GORILLAS and closing deals in the golf course instead of the lab. Well I am a goddam freakin’ monkey, and proud of it! And I want us monkeys to overthrow the slow, lazy, banana-eating gorillas who are inflating our daily jobs with mediocre experiences.

This is exactly the reason I founded Correlsense three years ago. I wanted to make a difference, or at least to try. And of course make a buck if I can while doing it Making a difference is by creating high quality software – the kind that you know is high quality from the moment you touch it. Making a difference is about solving problems to IT staff, it’s about educating your colleagues, your partners, your customers, and now you – on how you can become a better man in your job – by knowing what is bad practice and what is good, how to think when developing and maintaining an application, how you can solve hard performance problems instead of banging your head in the screen in front of you – I will tell you stories that will make you laugh and cry; I will make fun of those that deserve it and praise those that are worthy; I will expose the truth about code and architecture, about fuzzy algorithms that never work, about investors and top execs that try to look smart but just turn out dumb in the process, about arrogant programmers that don’t have a clue and much much more coming soon to a theater near you

You are right - I am terribly obnoxious - but I promise to be honest and at least tell the truth as I see it. No more, no less.

See you next time (if you dare), Lanir