Windows Azure Bests Amazon S3 in Storage Performance Shootout

Microsoft's Windows Azure BLOB storage performed significantly better in a shootout among five leading providers of public cloud infrastructure services, including last year's runaway winner Amazon Web Services.

Nasuni, a closely-held supplier of turnkey data protection appliances that use public Infrastructure as a Service (IaaS) providers' object storage repositories as backup and recovery targets, conducted the shootout for the second year in a row. While Nasuni officials said they conducted more exhaustive tests, such as by benchmarking a wider range of file sizes (from 1KB to 1GB), the company only compared five preferred IaaS providers -- Amazon, Google, Hewlett-Packard, Microsoft and Rackspace -- compared with 16 last year.

Among those holdovers that didn't make this year's cut were AT&T, Nirvanix and Peer1 Hosting. Nasuni decided to go with fewer providers this year because the company only wanted to test those they considered the most likely providers it would use as backup targets for its customers. The company currently uses Amazon exclusively for that purpose and last year's shootout results appear to have validated that choice.

"Amazon was just heads and shoulders ahead of the rest last year," said Conner Fee, Nasuni's director of marketing, who said he was shocked to see Microsoft turn the tables on Amazon this year. Nasuni rated the speed of reads, writes and deletes to Windows Azure BLOB services at 99.96 percent, while Amazon performed only at 68 percent.

Response times when reading, writing and deleting files to Windows Azure averaged a half-second, with Amazon dropping from first place to second, though still performing reasonably well, Fee said. Not faring as well were Rackspace, where response times were a second-and-a-half to two seconds. Fee said he was also surprised by Google's weak performance.

"This year, Microsoft's Windows Azure took a huge leap forward," Fee said. "It was incredibly surprising to us as we view this as a relative commodity space and we expect the experienced players to be out in frond. What we found is that Microsoft's investments in Azure that they've been talking about for a while gave them the opportunity to leapfrog Amazon."

Brad Calder, general manager for Windows Azure storage at Microsoft, spelled out those improvements in a November blog post, describing the company's next-generation storage architecture, called Gen2. Microsoft deployed what it calls a Flat Network Storage (FNS) architecture that enables high-bandwidth links to storage clients. It also replaces traditional hard disk drives (HDDs) with flash-based solid state drives (SSDs). Here's how Calder described FNS:

"This new network design and resulting bandwidth improvements allows us to support Windows Azure Virtual Machines, where we store VM persistent disks as durable network attached blobs in Windows Azure Storage. Additionally, the new network design enables scenarios such as MapReduce and HPC that can require significant bandwidth between compute and storage."

Given the reason Nasuni conducts these tests is to determine which cloud service providers to use, does this mean Nasuni will shift some or all of the data it backs up for its customers from Amazon to Windows Azure? Not so fast, according to Fee. "Amazon has always been our primary supplier and Azure our distant second," he said. "I think we'll see more opportunities to use them. Will this change this year? Maybe but probably not. There's a lot more widgets to be made before we're willing to jumps ship."

However in several conversations with Nasuni, officials describe IaaS providers as commodity providers of storage, equivalent to the role HDD vendors play to storage system vendors like EMC and NetApp. "We do this testing because we're constantly evaluating suppliers," he said. "We test, compare and benchmark because we always want to make sure we're using the best suppliers and want to make sure our customers have the best possible experience."

When speaking to Rackspace CTO John Engates about another matter, I asked if he had heard about his company's poor showing in the Nasuni tests (Fee said the company had shared the findings with all the providers but Rackspace hadn't responded). Engates, though familiar with last year's shootout, said he hadn't heard about this year's findings, hence he didn't want to comment.

But he did say it's tough to draw any conclusions based on any one set of tests or benchmarks. "It depends on what your customers are doing as to whether your cloud is perfect or not," Engates said. Much of the data stored in Rackspace Cloud Files tend to be large data types that are enhanced by its partner Akami's content delivery network (CDN), Engates said. Likewise, Fee received feedback from Amazon that suggested Amazon felt the tests were biased toward scenarios with lots of small files rather than large data types.

As it turns out, one of the reasons Microsoft's Windows Azure performed so well, Fee said, was that its architecture is optimized for large quantities of small files. "That's where Azure excelled," he said. "We based our tests from real-world customer data. It wasn't something we made up or can change. A lot of these guys were much better at handling larger files, and Azure exceeded well at small files and that really influenced the results."

Despite the strong showing for Windows Azure, Fee said he believes that with the investments all five companies are making, that all of them could be contenders moving forward. "It wouldn't surprise me to see a new leader next year," he said.

Posted by Jeffrey Schwartz on 02/19/2013 at 12:48 PM