Doubts Even Here: the point of disbelief

Posted by Ceri Davies Sat, 30 May 2009 21:29:00 GMT

It’s slightly late in the day, but as I’m currently picking apart what parts of VMware’s VI/vSphere stack are actually useful in *my* Real World, I’m going to respond to Chuck Hollis’ blog post Why Oracle Doesn’t Like VMware, even though it was posted nearly a month ago.

To state my position clearly, I agree with Chuck’s on 80% of his points, particularly with the general sentiment that it would be trivial for them to support VMware as a platform but for the fact that they may lose business if they do. However, his point #4, “VMware Functionality Competes With Oracle DBMS Features” is completely disingenuous.

There is *no* VMware functionality that competes with Oracle DBMS features, although VMware marketing would like you to believe that there is. Let’s break Chuck’s examples down.

Chuck says about RAC:

On one hand, we've got a multi-server configuration running
 Oracle's latest (and most expensive) RAC product.  It's doing load
 balancing, high availability, and making the hardware function as
 a giant pool.

Let’s think about this RAC setup. It will be serving the same database via multiple instances. The clients know about each instance and will choose another when the first fails

and about VMware:

On the other hand, we've got the same multi-server configuration
 running the much cheaper Oracle SE on VMware. 

It too is load balancing, offers high availability, and makes the
 hardware function as a single giant pool.  Many of the management
 tasks are handled quite well outside of Oracle's domain.

Now I’m thinking right now that Chuck doesn’t know, or more likely has conveniently forgotten, how RAC works and also seems to have made similar mistakes regarding VMware’s features. Either that, or he’s completely believing VMware’s hype, much as they’d both like us all to do.

VMware is not like RAC

A RAC configuration consists of multiple instances serving the same database. This isn’t even conceptually similar to multiple VMs running multiple Oracle SE instances with, necessarily, multiple databases.

“load-balancing”

I’ve only been administering tens of Oracle DBMS databases for 4 years, but I have no idea how one would load balance read/write clients across multiple database+instances in anything approaching a productive way. I’m going to go as far as to say that, at least generally, you can not.

“high availability”

I’ve already mentioned what I think about VMware HA. It doesn’t offer good protection against network failures and it doesn’t offer any protection against FC storage failures. In fact, you can’t even mirror FC storage with VMware unless you get your SAN to do it[1].

Additionally, even when a failure is detected, the only fix is to restart the VM and thereby the Oracle instance, implying the loss of in-flight transactions.

“hardware functioning as a single giant pool”

While both RAC and VMware can be argued as making the hardware they run on function as a single pool, these two pools have an entirely different purpose. The “RAC pool” will take a database query and do the same thing regardless of which node handles it, while the “VMware pool” will not (unless, of course, you happen to be running RAC in it).

There’s more: Fault Tolerance.

Chuck then goes on to say, regarding the VMware setup:

And VMware brings a few very cool features to the table
 that Oracle doesn't, like real fault tolerance[...]

That’s a little shocking.

While RAC can be used to provide high-availability, there are (probably a large proportion of) RAC customers who would be using RAC in order to scale past a single server. VMware Fault Tolerance doesn’t even allow you to scale past a single virtual CPU.

VMware Fault Tolerance has other issues, such as a limited list of supported CPUs, the requirement to reboot a VM on most of those CPUs in order to enable it (and since you have to turn FT off in order to patch the ESX cluster it’s running on, that’s a big deal), lack of support for thin-provisioned VMs, an inability to support physical RDM, the requirement for a dedicated gigabit NIC and some other more minor ones. However, I’m also worried that it might use the same algorithm to determine failure of the Primary VM as VMware HA does - the documentation certainly mentions heartbeats, file locking on shared storage to prevent mistaken failover and that “failover occurs if the host running the Primary VM fails” which is the same terminology as the VMware HA documentation uses. I wonder if the loss of FC connectivity or the VM network will cause a failover here?

Don’t believe the hype

So much as I agree with Chuck’s main point, I’m annoyed at the over-hyping of VMware’s availability features because I don’t believe that they’re as good as Chuck would like me to. At the end of the day, as an Oracle and VMware customer I’d love to see Oracle’s database supported in VMware, but I’m well aware of the limitations of both and need this kind of misleading information being disseminated like a hole in the head.

[1] Note that this means that if you use Raw Device Mapping (RDM) to try to mirror in your OS instead, you’re just as at risk as if you hadn’t bothered because the mapping is stored in the not-fault tolerant .vmx file).

Posted in , ,  | 1 comment | no trackbacks

VMware HA?

Posted by Ceri Davies Sun, 24 May 2009 22:07:00 GMT

As a platforms engineer responsible for design, build and maintenance I spend a lot of time thinking about failure. As good system administrators and system designers will know, this means evaluating the impact of the failure of every aspect of the system as well as assessing whether this impact is actually important to the SLA in order to design around it where necessary.

VMware HA

VMware HA, a feature of VMware’s Virtual Infrastructure and vSphere products (both of which I will refer to, incorrectly, as “ESX” below), believes that it has a space to fill here. VMware HA has two features:

  1. If an ESX host fails, VMware HA will power on the Virtual Machines that it was running on another host in the cluster, thereby restoring that VM to service;
  2. VMware HA can monitor if a guest operating system is still responding and perform the same action if it is not.

It does all this in a manner transparent to applications and guest operating systems and on paper it sounds really useful. It’s not a sub-second response to an outage, but for a large number of situations it would be good enough.

Problems

I’ve recently undertaken to design a VMware based infrastructure for my employer and have been performing a number of experiments with VMware HA. The results of this testing are not promising and I’m therefore blogging my findings with a number of possible outcomes in mind:

  1. Someone corrects me, points me to a reference and I fix my test setup;
  2. People stop telling me VMware HA is the answer to all ills.

As further background to the complaints below, a typical ESX installation uses at least three NICs; one (or more) for the ESX service console, one for VMotion and one (or more) for Virtual Machine traffic. VMware HA heartbeats are carried over the service console NICs.

VM Networks are not protected

My first finding was that disconnecting the NICs carrying Virtual Machine traffic (and thereby causing those VMs to become disconnected from the world outside the ESX host) does not cause VMware HA to initiate a failover, as the machine is still “in cluster”. Further to that, it doesn’t even cause a warning flag or even a log entry to be displayed in the high-level vCenter interface; if you specifically look at the Network Configuration page on that host, you see a small red “x” next to the NIC.

Storage networks are not protected

ESX can use SAS, iSCSI, NFS or Fibre Channel as a remote (i.e., not physically in the ESX host) storage backend. I’ve tested NFS and Fibre Channel, and losing all paths to the storage doesn’t cause a failover of the VMs. Again, no warning flag is displayed on the ESX host in vCenter. A warning is logged, but no error, and the warning is misleading - entries were logged regarding attempts to fail over to an alternative path, but no error was logged stating that these attempts had failed.

Heartbeat network failure does not trigger a failover (by default)

In the default configuration, even losing the heartbeat networks will not cause VMs to be failed over. This is because ESX uses file locks to indicate when a VM is powered on on another host in the cluster, and these locks remain in place even when a host becomes disconnected from the other hosts in the cluster as the default setting is to take no action when this occurs, although changing this setting to “Power Off VM” or “Shut Down VM” does allow the loss of heartbeats to cause a failover. Even when an ESX host panics, in my experience, a failover may not occur as the locks can still be held until the host is manually power-cycled.

HA?

The conclusion that I draw here is that VMware HA actually only protects service if an ESX host suddenly powers off. I really don’t feel that VMware HA deserves the “H” in its moniker and would prefer that it be renamed use VMware SEPCA (Slightly Elevated in Particular Circumstances Availability) from this point onwards; there’s a good history of renaming products and product features so this shouldn’t be too much turmoil. This would help people to think more about what they’re actually getting rather than assuming that they’re covered by ESX.

Posted in  | 3 comments | no trackbacks