Maintenance Sucks

Over the past week, I have been involved in multiple debates regarding the role of “maintenance” in establishing an effective software business. In one conversation with a key analyst at a top tier research firm, I listened to the analyst proclaim:

“Given the last 15 years of end user maintenance experience, customers have concluded that maintenance sucks. Avoid it if you can because something always breaks.”

Contrast that sentiment with Aneel Bhusri's (President of Workday) comments on a panel hosted by David Dobrin of B2banalysts:

“As a software vendor, the SaaS model is critical because it enables Workday to deliver updates and upgrades in a cost effective manner.”

If “maintenance sucks” and it should be avoided, why is Aneel so keen on efficient maintenance as a central tenant of the Workday business model? I believe I can bridge the gap between these statements by adding a couple of clarifying clauses:

“Given the last 15 years of end user maintenance experience [with legacy multi-platform server applications running on a bloated general purpose OS], customers have concluded that maintenance sucks [because, despite claims of certification, the patches from each vendor are never really tested in the exact configuration that we have deployed]. Avoid it [maintenance] if you can because something always breaks. [Unfortunately, customers that avoid maintenance create a miserable situation for a software vendor because the value of the application becomes marginalized while the vendor cost of maintaining legacy code on multiple platforms escalates]. As a vendor, the SaaS model [or a virtual appliance model] is critical because it enables Workday [or any other application vendor] to deliver updates and upgrades in a cost effective manner.”

When you pull these 2 sentiments together with a bit of editorial glue, they stick. SaaS is booming because it enables the vendor to deliver the application in a more efficient manner, and the customer gets application value without IT hassles. Salesforce.com is able to deliver their application to customers with just 8% of revenue for R&D compared with the more typical 16% that is standard for the application industry. I can only assume that Workday will achieve similar results due to the efficiency of maintaining a single code base running on a single platform for all customers.

Virtual appliances, done correctly, yield the same benefit. The vendor assumes responsibility for all software maintenance. The customer accepts maintenance (and presumably value) without hesitation because it is tested by the vendor before it is released to the customer – in the exact configuration used by the customer. The vendor and the customer get the added benefit (compared to SaaS) of flexibility regarding the network upon which the application is hosted. A virtual appliance is portable to any network that exposes a hypervisor host. The vendor can host it. The customer can host it. Or it can run on a service like Amazon's EC2 or IBM's pending Blue Cloud.

The concept must be gaining traction because the IT alarm sirens are shrieking:

WARNING! WARNING! VIRTUAL APPLIANCES ARE COMING! THEY MIGHT BE ON YOUR NETWORK ALREADY! THEY ARE BAD BECAUSE YOU LOSE CONTROL OVER THE TECHNOLOGY! THERE ARE SECURITY ISSUES! THERE ARE MANAGEMENT ISSUES! STOP THEM BEFORE USERS EXPERIENCE HOW EASY THEY ARE TO USE!

These are the same warnings that sounded regarding SaaS about 3 years ago. Anyone want to bet that the outcome will be different? My advice to the IT status quo – don't fight virtual appliances. Just like SaaS, customer value will trump IT control. Work with the emerging vendors to get what you need while giving your customers what they want -- application value without IT hassles. Maintenance doesn't have to suck.

Hypervisor Certification Crossroads

My last blog post, Certification “aka Some Assembly Required,” stirred up an interesting brew of fan mail. It seems that I wrote what lots of people have been thinking when I proclaimed that certification is a weak promise at best -- a hoax at worst -- with most customers spending 6X their license costs on installation, maintenance, and administration of “certified” software. The pity of this whole situation is that software vendors are also paying a high price for the myriad of customer preferences regarding middleware and operating systems. And the price is about to get a lot steeper.

Applications vendors will soon find themselves at a crossroads. They must either embrace virtual appliances (with certification to the hypervisors as the critical element), embrace SaaS ala Salesforce.com as their only distribution approach (where customers get no choices regarding infrastructure or even features for that matter), or brace for an enormous expense uptick in certification costs when they must begin including all of the hypervisor products as additional customer preference items required for testing.

The certification gauntlet that application vendors have historically run in order to deliver their application to the widest possible market is often referred to as “the matrix of pain.” For every element in the stack that might be a customer preference – OS, application server, web server, database – and for every release of these elements, the application vendor creates a row and a column. Each intersecting cell above the diagonal represents a configuration to be tested as part of the R&D expense associated with release engineering. Of course the testing never occurs in the exact configuration of elements for a given customer situation, but some testing is better than none.

And, theoretically, each maintenance update of each element requires re-testing. Of course, no application vendor actually does this testing because they simply cannot afford to do it. As it is, certification gobbles up 40 – 50% of R&D expenditures (not to mention the customer service burden associated with customer variability). Don't believe me? Why do you suppose salesforce.com spends 8 – 9% of revenue on R&D for a pretty rich CRM application when most software companies spend 15 – 18%? Salesforce.com only certifies to ONE infrastructure – their own datacenter infrastructure. And they bring every customer along with every release of software so they do not have the testing expense associated with maintaining legacy code on new platforms or vice versa.

Now, with hypervisors providing customers with so much value and flexibility as the layer that abstracts the hardware from the application, vendors are going to be pushed by their customers to certify their applications against all of the popular hypervisors. And, with such a hot category, there will be lots of technology development and lots of releases. That means lots of NEW columns and rows for the “matrix of pain.” If you thought 40 – 50% of your R&D budget was painful because it sucked the wind out of new feature development, wait until that number climbs to 60 – 70% when you throw in another layer for the hypervisor. Add some more to those costs when customers start demanding that you also support Amazon's EC2 and other similar services like IBM's Blue Cloud.

Had enough? Ready to cry “UNCLE?” Better get busy figuring out how to deliver your application as a virtual appliance. For those virtual appliances, you will STILL need to release and test against the hypervisor products -- VMware, Citrix' XenSource, Microsoft's pending Hyper-V -- along with all of the “cloud” computing services that are going to emerge to compete with Amazon's EC2. Better choose your virtual appliance infrastructure provider wisely to accommodate these customer choices in the market (shameless plug for rPath).

No doubt customers will initially object to some limitation of choice regarding the elements that support the application when you wrap it all up and deliver it with a bow on top as a virtual appliance, but these are the same customers that are going to drive you to embrace hypervisors in your testing matrix. Give them a tradeoff – virtual appliances that are cheap and easy to install, maintain, and administer, or the old fashioned way with a $250,000 per year support uplift to cover your expense for navigating their “matrix of pain.” Standing at this expense crossroads, I bet they make the same decision you should make as you stand at the hypervisor certification crossroads today -- “Let's give that virtual appliance a try.”