Is JeOS a Tonic for VM Sprawl?

It seems that everyone is worried about VM sprawl these days. When system capacity is easy to consume because the application is prepackaged as a virtual machine (a virtual appliance in the case of an ISV), the virtual infrastructure capacity is quickly gobbled up by those that have been waiting for IT to get around to provisioning systems for them. No more friction due to virtualization means no more waiting and wanting. It also leads to VM sprawl. But why is VM sprawl bad?

VM sprawl is bad because the scale of the management problem used to be throttled by the capital spending associated with the size of the infrastructure. Now, the scale of the management problem is equal to the true demand for application capacity as represented by the number of application images, or virtual machines, that get deployed. This scale factor associated with application images throws the old yardstick of X system admins per Y server machines out the window. What are we going to do to lower the work profile associated with so many new systems that now need to be managed?

I think at least part of the tonic for VM sprawl is the new acronym coined by Srinivas Krishnamurthi of VMware – JeOS. JeOS stands for Just enough OS and it is pronounced “juice.” In my mind, this liquid pronunciation is appropos given my view of its potential potency as a tonic for VM sprawl. A huge part of the burden of system management is the patching and associated regression testing for maintaining the security and functionality of the general purpose OS. If you can shrink the size of the OS by 90% (which is where we have measured the typical size for most applications built with our rBuilder technology) by eliminating any elements not required by the application, you can eliminate about 90% of the patching burden. More importantly, you eliminate the bigger burden of regression and stability testing that is coupled to the patching process.

With a JeOS approach, the number of virtual machines can theoretically grow by 10X the legacy approach without any impact to the cost structure associated with patching and testing the OS changes. I suspect a 10X reduction represents a real win for most shops as the OS patching and testing associated with security and infrastructure performance is a very sizable portion of their management spending. So if you are feeling the pangs associated with VM sprawl, I strongly suggest a healthy slug of JeOS each morning and once again in the afternoon to clear your system of the painful bloating that is brought on by virtualizing the general purpose OS.

VMware Strikes Back

The tech industry has been all abuzz lately about the competitive hullabaloo surrounding the new hypervisor technologies that are emerging to take on VMware's dominant hypervisor product. Microsoft launched Hyper-V with a party last week to upstage VMware's VMworld event this week. Red Hat purchased Qumranet to solidify its control of the KVM hypervisor technology. Now, VMware is striking back at the legacy OS vendors by labeling their new product category the Virtual Datacenter Operating System – a direct attack on the entrenched category of the general purpose operating system. The cold war of spies and covert operations to grab mindshare while outwardly promoting a message of peaceful co-existence has officially escalated to a hot war for the future architecture of the datacenter.

I, for one, am happy to see this rise in hostilities because I believe it will carry the industry to a much better place – and customers will be the primary beneficiary of the new approach. In the legacy datacenter, a general purpose operating system attempts to serve both the hardware infrastructure with device drivers while also serving the applications with system services. This approach has the disadvantage of artificially coupling applications to physical servers. Any attempt to move the application to another physical server typically requires that the configuration and validation process begin anew because it is extremely unlikely that the new server is absolutely identical to the previous one. This lack of flexibility leads to extreme overspending on capital equipment because an application with a period of low demand cannot relinquish its hardware resources to an application experiencing high demand. The hardware resources become application specific, and each application owner must size hardware capacity to meet peak demand. Server utilization in the datacenter averages 15 – 20%, and the general purpose OS is the culprit.

VMware has now declared that they offer an alternative approach to the general purpose operating system. The technology is not new, but marketing it under the category of an operating system is a very different tactic in this war for the datacenter. The conflict is now overt instead of covert, and this change was inevitable as VMware attempts to expand its footprint beyond its bread and butter business of Windows server consolidation and test lab operations. The new objective is the elastic datacenter and ultimately cloud computing.

The datacenter becomes elastic when applications are released by developers as coordinated sets of virtual machines (or virtual appliances in the case of a vendor release), each with Just Enough Operating System (JeOS or “juice”) attached to provide the system services required by the application. These applications can expand or contract on-demand because there is no onerous configuration process to ready the general purpose OS for a specific application. Instead, the hypervisor accepts the virtual machine and allocates it resources as specified by the OVF meta-data that is included with the image. Applications are up and running in a matter of seconds, and the process is totally repeatable to assure stability, security, and compliance as workloads scale, de-scale, and re-scale to meet the ever changing demand profiles of the enterprise application portfolio. Infrastructure can become a variable cost via this architecture because the scaling cycle can include hardware resources provided by third parties via a hypervisor layer – aka cloud computing as popularized by Amazon's Elastic Compute Cloud (EC2) via the Xen hypervisor.

The reason I label this new competitive tact by VMware as “warfare” is because the concept of a hypervisor as the infrastructure management layer with JeOS as the system services layer for the applications delivered as virtual machines destroys the value of the general purpose OS. If a hypervisor provides access to the infrastructure via device drivers, and applications receive system services from JeOS, and the flexibility of the datacenter improves, and the management of applications is simplified, and I can embrace cloud computing for variable cost infrastructure, why would I ever again buy a general purpose operating system? I won't. And customers won't either.

Unleash the dogs of war. Let's get to it so that we can all live happily ever after on the other side of history.

Red Hat oVirt(ly) targets VMware and Citrix

Red Hat announced last week that they are developing technology that will one day become a product to challenge the offerings from VMware and Citrix. The technology is showcased at a website sponsored and maintained by Red Hat as part of an “emerging technology” initiative. Aside from the fact that Red Hat is announcing technology projects and not products, the most noteworthy detail of this approach is the emergence of KVM (kernel-based virtual machine) as the hypervisor approach that Red Hat intends to back. Taking a page from the Microsoft HyperV playbook, Red Hat is claiming virtualization as a “feature” of the OS they already control in order to maintain their investment in the server distribution channel they have established with OEMs such as Dell, HP, IBM, Fujitsu, and others. Bare metal hypervisors and their associated infrastructure management frameworks are threatening the entrenched status quo of the operating system vendors, and Red Hat does not intend to “go gentle into that good night.”

Historically, a general purpose operating system performed two key functions: 1) provide drivers so that applications can access the hardware, and 2) provide system services (file system, libraries, etc.) to the applications. The trouble with this approach is that applications become artificially coupled to the hardware. Have you ever experienced pain upgrading hardware when the application that ran just fine on the old hardware no longer runs on the new hardware because of changes in the operating system? Do you find that you are continuously porting and testing applications just to upgrade hardware? Why? Because a one-size-fits-all general purpose operating system (OSFAGPOS) couples hardware support with application support due to the architecture of the product. Aside from the application portability issues, the OSFAGPOS approach also leads to the enormous management cost associated with bloating. Most applications run atop an operating system in the datacenter that is 10X the size actually required by the application, which leads to a patching nightmare.

To be fair to Red Hat, KVM could certainly be implemented and maintained by them as a skinny bare-metal hypervisor that only concerns itself with managing the hardware infrastructure. The Linux kernel is a fine provider of hardware support, and Red Hat has more Linux kernel expertise than any other vendor in the world. Their technical credibility in this space is terrific. The trick will be the marketing challenge of offering a product that has a very different value proposition than the OSFAGPOS. The goal of a bare-metal hypervisor is to support as many application OS variants as possible in order to enable application providers to choose the system software that works best for their application. This flexibility for the application flies in the face of the OSFAGPOS argument to “standardize” the OS to gain economies of scale in management and support. With OSFAGPOS, at least you know what systems need to be patched – all of them . . . all the time. Embracing a bare-metal product approach would mean that Red Hat also needs to face up to OSFAGPOS management challenges and embrace virtual appliance value for applications that run with Just enough OS (JeOS or "juice") optimized for their workload.

I believe that the separation of the duties of the historic OSFAGPOS architecture into two separate product domains - hypervisors for drivers and JeOS for application support - is inevitable. Separating hardware support from application support simply provides too many benefits. The popularity of both VMware and Amazon's emerging EC2 service are great examples. Watching both Red Hat and Microsoft navigate this change will be interesting. The marketing gurus at both companies will be strained to the breaking point as they “rage, rage, against the dying of the light.”

The Patching Dilemma

As virtual appliances pick up steam as an alternative approach to delivering SaaS value, I have seen a few analyst proclaim that the burden of patch delivery and management makes multi-tenancy via virtualization unattractive. They are correct . . . . if you attempt to build the customer virtual appliances using a legacy approach with a general purpose operating system without an integrated approach for lifecycle management. They are wrong if you consider a Just enough Operating System (JeOS or “juice”) approach with robust lifecycle management such as that offered by rPath. Let me provide an example.

rPath maintains a reference implementation of a full featured distribution of the Linux operating system as part of our rPath Appliance Platform offering. Compared with Red Hat and Novell, we probably offer about 80% of the software packages they provide as part of this reference. The remaining 20% represent desktop technology and other packages that do not matter for our target market, but are certainly important to their respective go-to-market strategies. As part of our commitment to maintain a full featured distribution, we have released about 200 security patches over the past 2 years. That is a lot of patches, but it is the reality of maintaining an OS. Keep this number in your head for a moment.

rPath also delivers our products to our customers as virtual appliances. Our ISV customers receive rBuilder and the rPath Appliance Platform as turn-key server applications completely managed by rPath on their network. Due to the unique packaging technology we pioneered, the operating system footprint to support rBuilder and the rPath Appliance Platform is about 50 Mb. When you use a JeOS architecture you eliminate any package that is not required by the application. Why is this important? Remember the 200 security patches released by rPath over the last 2 years? Only 3 were required to support our product implementation at our customers. That's correct – only 3.

Furthermore, because we deliver our patches to a specific implementation of the product (i.e. the customer did not assemble it themselves from multiple third party components, rendering clean application of patches virtually impossible), all of our customers received and applied the patches with no testing burden for them and no customer support burden for rPath.

Returning to the analysts that claim patching makes multi-tenancy via virtualization untenable, using a legacy approach with a general purpose operating system inside of snapshot virtual machine would ruin the economics of multi-tenancy via virtualization. With the rPath approach, coupled with rPath's technology for distributing patches to large numbers of systems with minimal administrator labor, you can host 66 customer virtual appliances for the same administrator effort as one virtual machine with the legacy model (3 patches vs. 200). And you avoid the expense of re-architecting and re-writing your code to support multi-tenancy – a VERY expensive proposition. And you avoid changing your business and sales model because customers can run the virtual appliances on-premise – but without the headaches of technology integration and multiple party maintenance management.

Virtual appliances deliver all of the value of SaaS to your customer base without all of the vendor hassles associated with changing your technology and changing your business model. However, just snapshotting an implementation of legacy components and ignoring the lifecycle management issues will not scale. Taking that approach would be crazy and unprofitable. rPath gives you the best of both worlds.

JeOS - Product or Architecture?

JeOS (pronounced “juice”) is a concept first described in writing by Srinivas Krishnamurti of VMware in this blog entry. With the pronouncement this week by Canonical that the Ubuntu distribution of Linux is now a JeOS product, I thought I would make the argument that JeOS is a packaging architecture, not an operating system product. I also believe that the hypervisor is the replacement for the product formerly labeled as the general purpose operating system.

Historically, and general purpose operating system served 2 important roles:

1) provide drivers that abstract the hardware and make the hardware resources and network available to the software applications, and

2) provide system services (file system, program libraries) to the applications such that they can execute their code in a predictable and efficient manner.

The 1st role above is now being relinquished to the hypervisor. Srinivas argues that the second role should be accomplished using JeOS. However, the actual configuration and components of JeOS, by definition, should vary considerably from application to application as each application needs different resources to operate efficiently. So, can JeOS be a product category sold in the same manner that general purpose operating systems were historically sold? As a “one size fits all” collection of general purpose utilities and libraries? Well, let's examine how the general purpose operating system is sold and determine if the same approach can be applied to the new JeOS category.

As background, I ran product marketing at Red Hat from early 1999 until midyear 2000, and I was the first enterprise sales leader for Red Hat. I was Vice President of North America sales for Red Hat from April 2001 until January 2005. During that time, we were very successful in growing sales of Red Hat Enterprise Linux by mimicking the Microsoft server OS product approach. Red Hat provided a stable release of operating system technology with a very broad set of utilities and libraries included. This release was accompanied by a commitment to five years (now seven, I think) of maintenance and regular updates for new hardware drivers, bug fixes, and security fixes.

It was also accompanied by a restriction that released Red Hat from its support obligation if the customer made any changes to the components that defined the operating system. Without this restriction, Red Hat could not possibly afford the engineering and customer service expense associated with supporting every customer's preference for utilities, program libraries, and system services. Further, the operating system itself required so many components to support so many components to support so many more components that changing any one of them often left the operating system in an unstable and unmaintainable state. This problem is often referred to by system administrators as “dependency hell.” But, since the operating system was often distributed by the hardware server providers as the mechanism to attach an application to the hardware, it was unavoidable that this restrictive model be strictly enforced to enable the industry to get on with shipping hardware that could be used by applications.

The restriction did not prevent customers from making changes to the operating system and subsequently seeking support from Red Hat. They simply absorbed the risk and much of the costs associated with the changes. If Red Hat turned them away because their changes did not conform to Red Hat's definition of the operating system, they soldiered on and did their best to make the operating system work for their applications. It never really occurred to them or to Red Hat that it might be a good idea to change the underlying packaging technology of the operating system to make it easy for the customer to change the things that did not work for them. Such a solution was unthinkable because it would leave the server hardware vendors with an undefined bill of materials. The server business is a high volume business that requires some level of standardization among the components to achieve price competitiveness through economies of scale. Changing the operating system for every server depending on the application workload it is intended to serve does not make any economic sense.

Now that the hypervisor is going to become the new operating system that supports the hardware, should the JeOS that supports any given application be a product with the same architecture as the legacy general purpose operating system? i.e. a collection of components defined by the operating system vendor as supportable and maintainable? so long as you don't change the assumptions the operating system vendor made when the collection was assembled and tested and released? so long as you don't change any of the components because that makes the collection unsupportable? How is that possible, when by definition, the collection of components that the application vendor is going to use is going to change depending on the needs of the application? Think about it.

JeOS cannot be an operating system product. JeOS must be a packaging architecture (and perhaps a testing architecture as well). JeOS must answer the question “What does the application need?” based upon the application it is presented. JeOS must then assemble around the application the components that represent the smallest possible closed loop set of components to host the application. It must then confirm that this “set” is reasonable based upon the testing scenario for the application (hence my comment above about a testing architecture). It must subsequently enable the integration of the various maintenance streams for the components with the server set to provide an elegant lifecycle experience for the application vendor and its customers. In this scenario, the application provider takes on a much broader responsibility for the support of the operating system, and at the surface level this can seem very scary.

On further inspection, however, the application vendor or their customer was already assuming this burden in the legacy model where the general purpose OS was modified on premise to support the application. With a fight through dependency hell followed by an ongoing war with the maintenance stream from the OS vendor (which was often ignored because it could not be reconciled with the needs of the application). With JeOS as a packaging architecture (instead of a “one size fits all” product), the component set that must be maintained is much smaller and much more intimately related to the application. By definition, the application vendor will be in a good position to determine and resolve problems given this tight definition and technical affinity with the application. No doubt they will still want the technical expertise of a vendor with deep operating system skills as a backstop, but the product they acquire from that vendor will look very different than the product historically labeled as a general purpose operating system. It will be a packaging architecture with a system software reference platform and a build/test methodology. Anything else defies logic in a world where every server has a hypervisor and every application arrives as a virtual appliance with JeOS.