Amazon EC2 versus Bare Metal and KVM? The inside story on what you thought you knew about EC2.

[Dean Bittner]

I've been using Amazon EC2, Amazon's Elastic Compute Cloud for a while now. I use EC2 primarily for testing where I prepare server and client environments and run tests. EC2's made vast computing resources available at commodity costs, and that's changed the way I do things, particularly by automating large numbers of client computers for load testing.

EC2 started humbly, with one type of instance, a very modest linux box. Then came more CPU, memory and disk, and of course Windows. Now, a wide array of machine types are available, boasting various numbers of CPU resources, IO capabilities, and storage.

That linux instance I used at the beginning doesn't seem as powerful, and not just by comparison. I also notice that, increasingly, I get highly variable and suspect results on my performance testing on EC2 instances. By variable, I mean that I get wide variance and large outliers. Also, the performance measures often don't line up with performance I am seeing in the field, on real machines.

My first response was to benchmark. Amazon instances actually benchmark favorably versus bare metal and other virtualization. I used the popular, although not free, passmark benchmark.

Then, very recently, I developed a test on my local machine, then ported to linux bare metal, a kvm virtual machine, and finally EC2. I found myself in possession of a test on three very interesting platforms: bare metal, kvm virtualization, and Amazon EC2. I could run the test on the three and compare, and that's just what I did, with some surprises.

The test runs some HTTP-based service calls between an automated client and a cloud-based server. During test development, I had noted some real discrepancies in how the test behaved on the cloud versus elsewhere.

Each bot in the test runs a basic automation with several calls to the server. An easy test is to run 250 bots concurrently, each running twice. All the HTTP traffic is SSL.

For EC2, I chose the m1.large instance. This is a 64-bit instance with two CPU's, high IO, and 7.5 G of RAM, plenty for this test. The bare metal box is a core2 quad @ 1.6GHz with 4 G RAM, and the KVM instance is hosted on this box as a KVM instance with 1 CPU and a little more than 1 G RAM.

The EC2 test fails miserably, with a 30% failure rate on the bots. Those that survive take an average of 30 seconds for each network operation round-trip, with a max of 130 seconds, a real high outlier. In isolation, this result would suggest a system in trouble, and one guess as to the system in trouble would be the server, but guess again.

The bare metal runs with an average timings of 3 seconds where EC2 ran 30, making it 10 times as fast. Gone too are the big outliers. The longest times are well withing a margin of acceptability, and most importantly, no errors were encountered.

There's one more surprise. The timings on the meager KVM instance were marginally faster than the bare metal. This seems to confirm KVM as an excellent platform for load and performance testing.

While the measurements vary slightly with more runs, the general trend holds. The Amazon m1.large instance just can't compare, either with the core2 quad at 1.6 GHz -- nor with a KVM virtual machine with 1 virtual CPU hosted on a 1.6 GHz core2 quad. And, KVM compares favorably with bare metal.

It is worthy of note that I can get more acceptable performance from the m1.xlarge and c1.xlarge instances; however, it seems out of line with any stretch of real-world comparison of machines with similar specs in terms of numbers of CPU's and IO performance. Also, I find that many EC2 machines indicate 'high IO' performance but perform very differently. For load and performance testing, this likely means that reliable results can only be had with the high-end of the Amazon EC2 instances.

I will also go hunting for someone who has KVM-based hosting, like VrSTORM.