News

2020 Cloud Report by Database Vendor Benchmarks Azure, AWS, GCP

Cockroach Labs has duplicated its benchmark testing of the top cloud computing platforms, finding that Amazon Web Services (AWS) no longer enjoys such a big price/performance edge over Google Cloud Platform (GCP).

The company, known for its open source CockroachDB that's described as a "SQL database for global cloud services," has been testing cloud platforms since internal testing showed better database performance on AWS and a more formalized study found it to be some 40 percent better than GCP.

"Not only is GCP more expensive than AWS, but it also achieves worse performance," the company concluded in late 2018. "We recommend using AWS for your most important workloads."

This year, for the 2020 Cloud Report (they skipped a version), Microsoft's Azure cloud platform was added to the benchmarking.

"We completed over 1,000 benchmark tests (including CPU, Network Throughput, Network Latency, Storage Read Performance, Storage Write Performance, and TPC-C), and found that the playing field looks a lot more equitable than it did last year," the company said in a Dec. 11 blog post. "Most notably, we saw that GCP has made noticeable improvements in the TPC-C benchmark such that all three clouds fall within the same relative zone for top-end performance."

Price per tpmC by Cloud Machine Type
[Click on image for larger view.] Price per tpmC by Cloud Machine Type (source: Cockroach Labs).

Because the company produces an online transaction processing (OLTP) database, it's primarily interested in testing transactional workloads, focusing on the TPC-C OLTP benchmark tool that simulates an e-commerce business to measure performance and efficiency. Most tests -- like one measuring throughput-per-minute type C (tpmC) (also known as the number of orders processed per minute) -- didn't show too much differentiation.

"Again, all three clouds come close on the cheapest price per tpmC. However, this year we see that the GCP n2-highcpu-16 offers the best performance per dollar in the tested machine types. If price is less of a concern, AWS is the best performer on throughput alone, but when is price not a factor?"

The report's conclusion provided more info. "GCP shows dramatic improvement in the 2020 Cloud Report edging out AWS and Azure on price per performance of TPC-C but slightly underperforming AWS and Azure on max tpmC available on a three node cluster," it said.

The database vendor -- which said it remains committed to providing a cloud-agnostic database -- also noted the importance of continuous testing.

"Setting up a highly performant configuration isn't always intuitive. It's also important to note that over the past couple years of testing, we've seen different cloud providers performance change drastically. Since these results fluctuate as the clouds adopt new hardware, it's important to regularly re-evaluate your configuration (and cloud vendor).

"CockroachDB remains committed to our stance as a cloud-agnostic database. We will continue to use AWS, Azure, GCP, and others for internal stability and performance testing. We also expect that these results will change over time as all three companies continue to invest in a modern infrastructure ecosystem."

Reproducible steps to duplicate the study are provided on a GitHub repository, and source code for the company's Roachprod open source provisioning system is available here for those interested in provisioning nodes exactly the same as the company did.

About the Author

David Ramel is an editor and writer at Converge 360.

Featured

Subscribe on YouTube